[00:41:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:11:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:58:52] (03open) 10kuldeepburjbhalaike: Revert "made wikisource url as optional" [toolforge-repos/bookitemizer] - 10https://gitlab.wikimedia.org/toolforge-repos/bookitemizer/-/merge_requests/4 [01:58:59] (03merge) 10kuldeepburjbhalaike: Revert "made wikisource url as optional" [toolforge-repos/bookitemizer] - 10https://gitlab.wikimedia.org/toolforge-repos/bookitemizer/-/merge_requests/4 [02:20:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:25:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:13:41] 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, 13Patch-For-Review: Set cloudvirt fleet to 'performance' governor - https://phabricator.wikimedia.org/T439729#12414633 (10Peter) I compared the performance and it made our CI test about 20% faster overall! Thank you so much @fgiunchedi ! The change diff... [05:48:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:58:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:13:54] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12414703 (10komla) [06:14:53] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Create scheduled job deployment tutorial - https://phabricator.wikimedia.org/T437901#12414704 (10komla) [06:39:12] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Create code change and redeploy tutorial - https://phabricator.wikimedia.org/T437900#12414725 (10komla) Formatted the command line examples to clearly show where they are being run(bastion or user laptop) [06:40:24] 06tools-infrastructure-team, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Test Platform: Add more CPUs to quibble jobs - https://phabricator.wikimedia.org/T439808#12414727 (10Peter) Lets add the instances when T440670 is done so the new instances use trixie. [06:55:43] 06tools-infrastructure-team: Decom old cloudcephosd hardware - https://phabricator.wikimedia.org/T440684 (10fgiunchedi) 03NEW [07:02:06] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Create scheduled job deployment tutorial - https://phabricator.wikimedia.org/T437901#12414786 (10komla) Formatted the command line examples to clearly indicate where they are running(bastion or user laptop) [07:02:37] 06tools-infrastructure-team: Enable perf_counters on cloudceph - https://phabricator.wikimedia.org/T440686 (10fgiunchedi) 03NEW [07:03:30] 10Tool-lingua-datasets: Create git repository - https://phabricator.wikimedia.org/T440687 (10Yug) 03NEW [07:04:33] 10Tool-lingua-datasets: Project init - https://phabricator.wikimedia.org/T440688 (10Yug) 03NEW [07:05:02] 10Tool-lingua-datasets: Project init - https://phabricator.wikimedia.org/T440688#12414828 (10Yug) [07:05:03] 10Tool-lingua-datasets: Create git repository - https://phabricator.wikimedia.org/T440687#12414827 (10Yug) [07:06:11] 10Tool-lingua-datasets: Create toolforge instance - https://phabricator.wikimedia.org/T440689 (10Yug) 03NEW [07:06:52] 10Tool-lingua-datasets: Create toolforge instance - https://phabricator.wikimedia.org/T440689#12414844 (10Yug) [07:14:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:24:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:27:38] 10Tool-lingua-datasets, 10Toolforge, 06tools-platform-team: Setup Lingua-datasets.toolforge.org with 400GB - https://phabricator.wikimedia.org/T440689#12414918 (10Yug) [07:28:30] 10Tool-lingua-datasets, 10Toolforge, 06tools-platform-team: Setup Lingua-datasets.toolforge.org with 400GB - https://phabricator.wikimedia.org/T440689#12414926 (10Yug) [07:28:46] 10Tool-lingua-datasets, 10Toolforge, 06tools-platform-team: Setup Lingua-datasets.toolforge.org with 400GB - https://phabricator.wikimedia.org/T440689#12414928 (10Yug) [07:31:34] 10Tool-lingua-datasets, 10Toolforge, 06tools-platform-team: Setup Lingua-datasets.toolforge.org with 400GB - https://phabricator.wikimedia.org/T440689#12414932 (10Yug) [07:34:07] 10Cloud-VPS (Project-requests), 10Language-Diversity-Hub: Request creation of VPS project : LinguaData - https://phabricator.wikimedia.org/T440447#12414937 (10Yug) Hello @Andrew , thank for your feedback. I took a pause of a few hours to reply. [EDIT]: Following your comments, I created a toolforge project w... [07:36:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:41:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:42:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:52:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:59:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:04:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:12:22] (03PS1) 10Brouberol: ceph::admin: update CI secrets to reflect recent changes [labs/private] - 10https://gerrit.wikimedia.org/r/1353839 [08:12:32] (03CR) 10Brouberol: [C:03+2] ceph::admin: update CI secrets to reflect recent changes [labs/private] - 10https://gerrit.wikimedia.org/r/1353839 (owner: 10Brouberol) [08:12:34] (03CR) 10Brouberol: [V:03+2 C:03+2] ceph::admin: update CI secrets to reflect recent changes [labs/private] - 10https://gerrit.wikimedia.org/r/1353839 (owner: 10Brouberol) [08:36:52] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Engineering, and 4 others: Set up x1 replication to Wiki Replicas - https://phabricator.wikimedia.org/T395881#12415133 (10Gehel) [08:38:05] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Engineering, and 4 others: Create wiki replicas views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#12415161 (10Gehel) [08:38:14] 10Cloud Services Proposals, 10Data-Services, 06tools-platform-team, 06Data-Persistence, and 2 others: Decision request - Who runs wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607#12415163 (10Gehel) [08:48:56] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 10BetaFeatures, and 6 others: Create view for betafeatures_user_counts table in wiki replicas - https://phabricator.wikimedia.org/T402145#12415401 (10Gehel) [09:04:09] (03update) 10fnegri: Update deps for K8s 1.34 [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/37 (https://phabricator.wikimedia.org/T414595) (owner: 10andrew) [09:04:13] (03update) 10fnegri: Update deps for K8s 1.34 [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/37 (https://phabricator.wikimedia.org/T414595) (owner: 10andrew) [09:04:20] (03update) 10fnegri: Update deps for K8s 1.34 [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/37 (https://phabricator.wikimedia.org/T414595) (owner: 10andrew) [10:24:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:43:13] 10Toolforge, 06tools-platform-team: Move Trove database back to ToolsDB - https://phabricator.wikimedia.org/T440700#12415757 (10Bugreporter) [10:47:48] 10Cloud-VPS, 06tools-infrastructure-team: Move Trove database back to ToolsDB - https://phabricator.wikimedia.org/T440700#12415767 (10Bugreporter) 05Open→03Stalled You can create a ToolsDB database yourself and try migrating. Mark as stalled until glamtools stop using Trove (so Trove quota can be freed). [11:19:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:20:51] 10Cloud-VPS, 06tools-infrastructure-team: Move Trove database back to ToolsDB - https://phabricator.wikimedia.org/T440700#12416065 (10fgiunchedi) +1 to what @Bugreporter said. You can move the database to toolsdb yourself, please reach out here once that's done and we'll proceed to remove the trove DB [13:02:25] 10Tool-lingua-datasets, 10Toolforge (Quota-requests), 06tools-platform-team: Setup Lingua-datasets.toolforge.org with 400GB - https://phabricator.wikimedia.org/T440689#12416175 (10JJMC89) [13:06:56] 10Cloud-VPS, 06tools-infrastructure-team: Move Trove database back to ToolsDB - https://phabricator.wikimedia.org/T440700#12416188 (10magnusmanske) DB has been moved, you are clear to remove the trove DB. [13:07:12] 10Toolforge, 06tools-platform-team: Mix'n'match: Temporary DB size increase to reduce DB size - https://phabricator.wikimedia.org/T440708 (10magnusmanske) 03NEW [13:28:39] 10Cloud-VPS, 06tools-infrastructure-team: Move Trove database back to ToolsDB - https://phabricator.wikimedia.org/T440700#12416240 (10Bugreporter) 05Stalled→03Open [13:39:37] 10Toolforge, 10Tools, 06tools-platform-team: Mix'n'match: Temporary DB size increase to reduce DB size - https://phabricator.wikimedia.org/T440708#12416261 (10fnegri) a:03Magnus @magnusmanske thanks for opening this task! I think it's ok to go with the temporary table approach, if possible please start it... [13:50:51] 10Toolforge, 10Tools, 06tools-platform-team: Mix'n'match: Temporary DB size increase to reduce DB size - https://phabricator.wikimedia.org/T440708#12416324 (10magnusmanske) Great. The plan is to create a new `entry_v2` table with indices (can't add them later easily), same fields, but page-compressed. Then,... [14:09:58] (03update) 10andrew: Update deps for K8s 1.34 [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/37 (https://phabricator.wikimedia.org/T414595) [14:18:07] 06tools-platform-team, 10Toolhub, 13Patch-For-Review: [Toolhub] Remove the legacy CNAME endpoints for the URL downloaders - https://phabricator.wikimedia.org/T439732#12416393 (10fnegri) I deployed the change following [How to deploy](https://wikitech.wikimedia.org/wiki/Toolhub.wikimedia.org#How_to_deploy).... [14:22:41] 06tools-platform-team, 10Toolhub, 13Patch-For-Review: [Toolhub] Remove the legacy CNAME endpoints for the URL downloaders - https://phabricator.wikimedia.org/T439732#12416400 (10fnegri) I think I got the patch wrong :/ I put `http_proxy: http://urldownloader.svc.eqiad.wikimedia.org` instead of `urldownloader... [14:34:14] 10Tool-wikinewsie: Recent sources for Wikinewsie - https://phabricator.wikimedia.org/T433528#12416437 (10WebRunner95) The [[ https://wikisignals-article-analysis.toolforge.org/ | WikiSignals Article Analysis ]] tool ([[ https://gitlab.wikimedia.org/toolforge-repos/wikisignals-article-analysis | GitLab repo link... [14:56:58] 10Tool-wikinewsie: Recent sources for Wikinewsie - https://phabricator.wikimedia.org/T433528#12416505 (10WebRunner95) The WikiSignals Article Analysis UI and API produce a table of each unique citation and metadata about the source. It's quite extensive, but for Wikinewsie, we could narrow it down to a few field... [14:59:16] 10Toolforge, 10Tools, 06tools-platform-team: Mix'n'match: Temporary DB size increase to reduce DB size - https://phabricator.wikimedia.org/T440708#12416516 (10magnusmanske) Turns out I don't have privileges to create triggers myself. I'll add them here next week for you to create when everything is ready to go. [15:03:59] 06tools-platform-team, 10Toolhub, 13Patch-For-Review: [Toolhub] Remove the legacy CNAME endpoints for the URL downloaders - https://phabricator.wikimedia.org/T439732#12416559 (10fnegri) 05In progress→03Resolved Fixed the URL and the crawler is now working again, using the new proxy URL (`urldownloade... [15:04:21] 10Toolforge, 06tools-platform-team: Can't retrieve logs for wsstats tool - https://phabricator.wikimedia.org/T440714 (10Soda) 03NEW [15:10:15] !log tools.cluebot3 Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/37949440447 (https://github.com/cluebotng/component-configs/commits/902367266d74fd0d5bf2343e1502e8e1eefc330c) [15:10:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [15:16:15] 10Cloud-VPS, 06tools-infrastructure-team: Move Trove database back to ToolsDB - https://phabricator.wikimedia.org/T440700#12416625 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi All done! [15:24:42] (03open) 10andrew: wmcs-k8s-metrics: bump chart version [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1463 (https://phabricator.wikimedia.org/T414595) [15:25:05] (03update) 10andrew: wmcs-k8s-metrics: bump chart version [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1463 (https://phabricator.wikimedia.org/T414595) [15:33:48] 10Cloud-VPS, 06tools-infrastructure-team, 06Test Platform (San Miguel de Allende 31): Test/benchmark virtio multiqueue for integration-cache-proxy - https://phabricator.wikimedia.org/T440418#12416710 (10fgiunchedi) [15:35:45] 10Cloud-VPS, 06tools-infrastructure-team, 06Test Platform (San Miguel de Allende 31): Test/benchmark virtio multiqueue for integration-cache-proxy - https://phabricator.wikimedia.org/T440418#12416712 (10fgiunchedi) >>! In T440418#12414030, @Mhurd wrote: > Sure, I'll add the setup to the description. One thin... [15:46:37] 10Tool-lingua-datasets, 10Toolforge (Quota-requests), 06tools-platform-team: Setup Lingua-datasets.toolforge.org with 400GB - https://phabricator.wikimedia.org/T440689#12416758 (10fgiunchedi) Thank you for reaching out. I'm +1 generally on the request, I have some more questions re: storage though * What is... [15:48:01] 10Toolforge, 06tools-platform-team: logs-api: GET /tool/{tool}/logs fails with a 500 when a stream has no "job_name" - https://phabricator.wikimedia.org/T440718 (10fnegri) 03NEW [15:48:55] 10Toolforge, 06tools-platform-team: [logs-api] API returns 500 when a stream has no "job_name" - https://phabricator.wikimedia.org/T440718#12416775 (10fnegri) [15:48:56] 10Cloud-VPS, 06tools-infrastructure-team: Move Trove database back to ToolsDB - https://phabricator.wikimedia.org/T440700#12416776 (10Andrew) We should remove the project entirely, right? I think it only existed to contain the db. [15:50:09] 10Cloud-VPS, 06tools-infrastructure-team: Move Trove database back to ToolsDB - https://phabricator.wikimedia.org/T440700#12416784 (10magnusmanske) yes, the rest if on toolforge/toolsdb. Thanks both! [15:52:56] 10Toolforge, 06tools-platform-team: [logs-api] Using -f on a job with no recent logs fails with Internal Server Error - https://phabricator.wikimedia.org/T429265#12416787 (10fnegri) [15:54:41] 10Toolforge, 06tools-platform-team: [logs-api] Using -f on a job with no recent logs fails with Internal Server Error - https://phabricator.wikimedia.org/T429265#12416790 (10fnegri) This was reported today by @Soda in [#wikimedia-cloud](https://wm-bot.wmcloud.org/logs/%23wikimedia-cloud/20261009.txt), on their... [15:57:31] !log andrew@cloudcumin1001 baglama2 START - Cookbook wmcs.vps.delete_project for project baglama2 in eqiad1 (T440700) [15:57:35] T440700: Move Trove database back to ToolsDB - https://phabricator.wikimedia.org/T440700 [15:57:54] (03update) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: delete project baglama2 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/367 (https://phabricator.wikimedia.org/T440700) [15:57:56] (03open) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: delete project baglama2 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/367 (https://phabricator.wikimedia.org/T440700) [15:59:30] (03merge) 10andrew: projects: delete project baglama2 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/367 (https://phabricator.wikimedia.org/T440700) (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [16:01:14] !log andrew@cloudcumin1001 baglama2 END (PASS) - Cookbook wmcs.vps.delete_project (exit_code=0) for project baglama2 in eqiad1 (T440700) [16:05:48] (03approved) 10fnegri: Update deps for K8s 1.34 [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/37 (https://phabricator.wikimedia.org/T414595) (owner: 10andrew) [16:05:52] (03update) 10fnegri: Update deps for K8s 1.34 [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/37 (https://phabricator.wikimedia.org/T414595) (owner: 10andrew) [16:08:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:14:10] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: ST5.4.3 Implement Toolforge UI MVP - https://phabricator.wikimedia.org/T440589#12416897 (10TLepage-WMF) [16:22:41] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: ST5.4.3 Implement Toolforge UI MVP - https://phabricator.wikimedia.org/T440589#12416966 (10TLepage-WMF) [16:27:54] 10Tool-Pageviews: Add clickable pie slice in the Traffic graphic - https://phabricator.wikimedia.org/T440726 (10Sadads) 03NEW [16:31:28] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: ST5.4.2 Toolforge Alerting System - https://phabricator.wikimedia.org/T432860#12417003 (10fnegri) //Note: we are experimenting with cross-posting to Phabricator the weekly report that previously was only posted to Asana, where only few people have acc... [16:35:36] 10Toolforge, 06tools-platform-team, 07OKR-Work: [components-api] send deployment events to the logs-api system-event log endpoint - https://phabricator.wikimedia.org/T432571#12417046 (10TLepage-WMF) 05In progress→03Resolved [16:38:50] (03update) 10andrew: Update deps for K8s 1.33 [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/37 (https://phabricator.wikimedia.org/T414595) [16:48:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:49:59] (03update) 10andrew: kube-state-metrics: bump chart version [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1463 (https://phabricator.wikimedia.org/T414595) [16:50:32] (03update) 10andrew: kube-state-metrics: bump chart version [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1463 (https://phabricator.wikimedia.org/T414595) [16:50:56] (03update) 10fnegri: add pagination [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/75 (https://phabricator.wikimedia.org/T440294) (owner: 10dcaro) [17:07:59] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolforge-deploy] Refactor wmcs-k8s-metrics component - https://phabricator.wikimedia.org/T388382#12417160 (10fnegri) [17:30:17] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.1.3 (T414595) [17:30:20] !log andrew@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.1.3 (T414595) [17:30:22] T414595: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.34 - https://phabricator.wikimedia.org/T414595 [17:31:23] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 (T414595) [17:31:24] !log andrew@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 (T414595) [17:32:30] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 (T414595) [17:32:31] !log andrew@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 (T414595) [17:32:56] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 (T414595) [17:32:57] !log andrew@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 (T414595) [17:33:43] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 (T414595) [17:33:44] !log andrew@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 (T414595) [17:35:16] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 (T414595) [17:35:17] !log andrew@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 (T414595) [17:35:24] T414595: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.34 - https://phabricator.wikimedia.org/T414595 [17:37:19] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 (T414595) [17:37:21] !log andrew@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 (T414595) [17:37:45] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 (T414595) [17:37:46] !log andrew@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 (T414595) [17:37:52] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 (T414595) [17:37:54] !log andrew@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 (T414595) [17:49:57] 10Toolforge, 06tools-platform-team, 07OKR-Work: api-gateway: Support bearer-token auth for Striker - https://phabricator.wikimedia.org/T440736 (10TLepage-WMF) 03NEW [17:50:28] 10Toolforge, 06tools-platform-team, 07OKR-Work: api-gateway: Support bearer-token auth for Striker - https://phabricator.wikimedia.org/T440736#12417352 (10TLepage-WMF) [17:50:29] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: ST5.4.3 Implement Toolforge UI MVP - https://phabricator.wikimedia.org/T440589#12417353 (10TLepage-WMF) [18:10:32] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 (T414595) [18:10:33] !log andrew@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 (T414595) [18:10:37] T414595: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.34 - https://phabricator.wikimedia.org/T414595 [18:23:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:08:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:36:55] (03update) 10andrew: kube-state-metrics: bump chart version [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1463 (https://phabricator.wikimedia.org/T414595) [20:19:34] 14cloud-services-team (Kanban), 10Cloud-VPS, 06tools-infrastructure-team: Create banner image for Wikimedia Cloud VPS - https://phabricator.wikimedia.org/T177442#12418136 (10LucasWerkmeister) I also added the file to Commons now (since I was confused why I couldn’t find it there and @Nemoralis was able t... [20:29:24] FIRING: SystemdUnitFailed: kiwix-mirror-update.service on clouddumps1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:16:31] 06tools-infrastructure-team: Upgrade cloudcumin hosts to Trixie - https://phabricator.wikimedia.org/T436667#12418462 (10Volans) I realized I never hit submit to my update comment with: `cloudcumin2001` has been upgraded, this was on Oct. 5th. I'll take care of cloudcumin1001 next week. [22:43:57] (03CR) 10Dzahn: "likely abandonable but I would not do that on someone elses's change, it feels wrong. though technically "restore" is cheap too." [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1343614 (https://phabricator.wikimedia.org/T438777) (owner: 10Krinkle)