[00:35:01] (03update) 10raymond-ndibe: [jobs-api] check services diff [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/158 (https://phabricator.wikimedia.org/T392717) [00:35:16] (03update) 10raymond-ndibe: [jobs-api] check services diff [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/158 (https://phabricator.wikimedia.org/T392717) [00:49:27] (03update) 10raymond-ndibe: [jobs-api] check services diff [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/158 (https://phabricator.wikimedia.org/T392717) [00:54:00] RESOLVED: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [00:54:30] FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [00:58:40] (03update) 10raymond-ndibe: [jobs-api] check services diff [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/158 (https://phabricator.wikimedia.org/T392717) [01:02:17] (03update) 10raymond-ndibe: [jobs-api] check services diff [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/158 (https://phabricator.wikimedia.org/T392717) [01:04:30] RESOLVED: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [01:09:16] (03update) 10raymond-ndibe: [jobs-api] check services diff [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/158 (https://phabricator.wikimedia.org/T392717) [01:09:30] FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [01:14:32] (03update) 10raymond-ndibe: [jobs-api] check services diff [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/158 (https://phabricator.wikimedia.org/T392717) [01:59:30] RESOLVED: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [02:23:16] (03update) 10raymond-ndibe: [jobs-api] check services diff [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/158 (https://phabricator.wikimedia.org/T392717) [03:49:34] 06cloud-services-team, 10Toolforge (Toolforge iteration 19): [functional-tests,deploy,cookbook] Run only selected tests when deploying a component - https://phabricator.wikimedia.org/T381011#10778792 (10Raymond_Ndibe) Thanks David for the comments! helped plenty [04:04:51] 06cloud-services-team, 10Toolforge (Toolforge iteration 19): [functional-tests,deploy,cookbook] Run only selected tests when deploying a component - https://phabricator.wikimedia.org/T381011#10778805 (10Raymond_Ndibe) > builds-builder: builds-api It still feels like we need to test maintain-harbor (as a proxy... [04:19:59] 06cloud-services-team, 10Toolforge (Toolforge iteration 19): [functional-tests,deploy,cookbook] Run only selected tests when deploying a component - https://phabricator.wikimedia.org/T381011#10778831 (10Raymond_Ndibe) Also I think there are some kyverno thing in `maintain-kubeusers` (vague recollection can't r... [07:03:18] 06cloud-services-team, 10Toolforge (Toolforge iteration 19): [functional-tests,deploy,cookbook] Run only selected tests when deploying a component - https://phabricator.wikimedia.org/T381011#10779030 (10dcaro) > It still feels like we need to test maintain-harbor (as a proxy to testing harbor itself) when buil... [07:32:52] (03open) 10taavi: tests: Update example K8s API server hostname [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/78 (https://phabricator.wikimedia.org/T262562) [07:34:23] (03update) 10taavi: tests: Update example K8s API server hostname [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/78 (https://phabricator.wikimedia.org/T262562) [07:37:05] (03PS2) 10Klausman: thanos/swift: at pseudo secrets for mint_ro [labs/private] - 10https://gerrit.wikimedia.org/r/1140112 [07:37:45] (03open) 10taavi: utils: Update auto-update commit messages [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/50 [07:47:48] 10Tool-inteGraality, 10Wikibase (3rd party installations), 06Wikibase Suite Team, 07Product-Feature: integraality for Wikibases? - https://phabricator.wikimedia.org/T294892#10779177 (10Addshore) It looks like this task was sucked into their workflows, they decided they didn't want to spend time on it, and... [07:50:22] (03approved) 10dcaro: utils: Update auto-update commit messages [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/50 (owner: 10taavi) [07:52:04] (03update) 10taavi: utils: Update auto-update commit messages [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/50 [07:52:08] (03merge) 10taavi: utils: Update auto-update commit messages [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/50 [07:57:40] RESOLVED: [2x] ProjectProxyMainProxyInstanceDown: Proxy on proxy-5 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/MainProxyInstanceDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProjectProxyMainProxyInstanceDown [08:07:11] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [08:07:20] 06cloud-services-team: Temperature Inlet Temp issue on clouddumps1001:9290 - https://phabricator.wikimedia.org/T392985 (10phaultfinder) 03NEW [08:11:10] FIRING: ProjectProxyMainProxyInstanceDown: Proxy on proxy-5 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/MainProxyInstanceDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProjectProxyMainProxyInstanceDown [08:12:11] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [08:14:26] (03approved) 10aborrero: Moved dns provider block for toolsbeta [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/12 (owner: 10chuckonwumelu) [08:14:38] (03merge) 10aborrero: Moved dns provider block for toolsbeta [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/12 (owner: 10chuckonwumelu) [08:15:08] (03update) 10aborrero: Importing tools dns resources [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/13 (owner: 10chuckonwumelu) [08:16:10] RESOLVED: ProjectProxyMainProxyInstanceDown: Proxy on proxy-5 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/MainProxyInstanceDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProjectProxyMainProxyInstanceDown [08:18:41] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [08:34:01] (03approved) 10dcaro: tests: Update example K8s API server hostname [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/78 (https://phabricator.wikimedia.org/T262562) (owner: 10taavi) [08:34:16] (03merge) 10taavi: tests: Update example K8s API server hostname [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/78 (https://phabricator.wikimedia.org/T262562) [08:36:33] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [infra] Fix the mis-named k8s service in tools and toolsbeta projects - https://phabricator.wikimedia.org/T262562#10779355 (10taavi) [08:43:41] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [08:48:12] 10Tool-erinnermich: [ErinnerMichBot] Script stops with error when date parameter contains blanks - https://phabricator.wikimedia.org/T392990 (10Tkarcher) 03NEW [08:57:30] 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: Enable IPv6 for the Cloud VPS web proxy - https://phabricator.wikimedia.org/T379175#10779411 (10taavi) [09:25:08] 06cloud-services-team: Temperature Inlet Temp issue on clouddumps1001:9290 - https://phabricator.wikimedia.org/T392985#10779530 (10fnegri) →14Duplicate dup:03T383723 [09:25:11] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [09:25:12] 10cloud-services-team (FY2024/2025-Q3-Q4), 06DC-Ops, 10ops-eqiad, 06SRE: Temperature Inlet Temp issue on clouddumps1001:9290 - https://phabricator.wikimedia.org/T383723#10779532 (10fnegri) [09:25:16] 06cloud-services-team: Temperature Inlet Temp issue on clouddumps1001:9290 - https://phabricator.wikimedia.org/T392996 (10phaultfinder) 03NEW [09:26:24] 10Cloud-VPS (Project-requests): Request creation of MissSearch VPS project - https://phabricator.wikimedia.org/T392903#10779548 (10Galr600) Hi @dcaro, Thank you for your response! Our current estimated storage requirement for MissSearch is approximately ~250GB. This is the minimum needed at this stage to contin... [09:30:38] 06cloud-services-team, 10Horizon: Allow encrypting connection from the shared Cloud VPS web proxy to the backend - https://phabricator.wikimedia.org/T274386#10779578 (10taavi) >>! In T274386#6819605, @bd808 wrote: > Untested idea for multi-hop: Based on https://nginx.org/en/docs/http/ngx_http_proxy_module.htm... [09:31:24] (03open) 10aborrero: kubeconfig: support updating the file [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/68 (https://phabricator.wikimedia.org/T262562) [09:34:04] 10Tool-inteGraality, 10Wikibase (3rd party installations), 06Wikibase Suite Team, 07Product-Feature: integraality for Wikibases? - https://phabricator.wikimedia.org/T294892#10779595 (10Olea) >>! In T294892#7501886, @JeanFred wrote: > @Addshore From a product perspective, would it make sense to ship integra... [09:34:33] 10Tool-inteGraality, 10VerySmallGLAM, 10Wikibase (3rd party installations), 06Wikibase Suite Team, 07Product-Feature: integraality for Wikibases? - https://phabricator.wikimedia.org/T294892#10779596 (10Olea) [09:35:53] 10cloud-services-team (FY2024/2025-Q3-Q4), 06DC-Ops, 10ops-eqiad, 06SRE: Temperature Inlet Temp issue on clouddumps1001:9290 - https://phabricator.wikimedia.org/T383723#10779603 (10fnegri) 05Resolved→03Open Reopening as unfortunately the alert is still flapping. It looks like the whole rack's temperatu... [09:38:36] 10Tool-inteGraality, 10VerySmallGLAM, 10Wikibase (3rd party installations), 06Wikibase Suite Team, 07Product-Feature: integraality for Wikibases? - https://phabricator.wikimedia.org/T294892#10779614 (10Olea) >>! In T294892#10779177, @Addshore wrote: > Ideally all Wikibases would run wikibase manifest ht... [09:45:11] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [09:53:37] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: [wikireplicas] Create views for new wiki nupwiki - https://phabricator.wikimedia.org/T390714#10779654 (10fnegri) 05Open→03In progress Ack, thanks. [10:03:02] (03open) 10raymond-ndibe: [toolforge-deploy] run specific tests on deploy [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/755 (https://phabricator.wikimedia.org/T381011) [10:09:13] (03open) 10dcaro: foxtrot_ldap: move install into foxtrot_ldap role [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/238 [10:14:25] (03update) 10raymond-ndibe: [toolforge-deploy] run specific tests on deploy [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/755 (https://phabricator.wikimedia.org/T381011) [10:17:42] 10Cloud-VPS (Quota-requests), 06Moderator-Tools-Team, 10Wikilink-Tool: Request to increase object storage capacity - Wikilink project - https://phabricator.wikimedia.org/T392870#10779719 (10taavi) Could you please update the request to have some concrete numbers - i.e. how much space and how many objects in... [10:18:11] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: [wikireplicas] Create views for new wiki nupwiki - https://phabricator.wikimedia.org/T390714#10779720 (10fnegri) The cookbook worked fine on an-redacteddb1001, but failed on clouddbs with: ` pymysql.err.OperationalError: (1044, "Access denied for use... [10:22:39] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: [wikireplicas] Create views for new wiki nupwiki - https://phabricator.wikimedia.org/T390714#10779722 (10fnegri) > the nupwiki_p table has been created on an-redacteddb1001.eqiad.wmnet I think the nupwiki_p database was created only on an-redacteddb1... [10:25:54] (03update) 10raymond-ndibe: [toolforge-deploy] run specific tests on deploy [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/755 (https://phabricator.wikimedia.org/T381011) [10:40:11] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [10:48:08] 10Tool-inteGraality, 10VerySmallGLAM, 10Wikibase (3rd party installations), 06Wikibase Suite Team, 07Product-Feature: integraality for Wikibases? - https://phabricator.wikimedia.org/T294892#10779801 (10Addshore) @Olea yes, wikibase manifest is likely highly relevant for {T391828} in terms of discovering... [10:48:37] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services, 06Data-Persistence, 13Patch-For-Review: wikireplicas: maintain-views should not create _p databases - https://phabricator.wikimedia.org/T392105#10779803 (10fnegri) It looks like the `CREATE DATABASE` part of the `maintain-views` cookbook is not ev... [10:55:01] (03update) 10aborrero: kubecerts: stop renewing certs on a random basis [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/69 (https://phabricator.wikimedia.org/T365681) [10:55:09] (03open) 10aborrero: kubecerts: stop renewing certs on a random basis [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/69 (https://phabricator.wikimedia.org/T365681) [10:56:18] (03update) 10aborrero: kubecerts: stop renewing certs on a random basis [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/69 (https://phabricator.wikimedia.org/T365681) [11:23:47] (03update) 10dcaro: crds: use the api name in the crd name [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/68 (https://phabricator.wikimedia.org/T386829) [11:25:17] 10Tool-inteGraality, 10VerySmallGLAM, 10Wikibase (3rd party installations), 06Wikibase Suite Team, 07Product-Feature: integraality for Wikibases? - https://phabricator.wikimedia.org/T294892#10779885 (10Olea) >>! In T294892#10779801, @Addshore wrote: > Everything on that API is either dynamically calcula... [11:34:48] 10Tool-inteGraality, 10VerySmallGLAM, 10Wikibase (3rd party installations), 06Wikibase Suite Team, 07Product-Feature: integraality for Wikibases? - https://phabricator.wikimedia.org/T294892#10779916 (10jon_amar-WMDE) @JeanFred yikes! my sincere apologies for closing this one. @Addshore is indeed correct... [11:47:36] 10Tool-inteGraality, 10VerySmallGLAM, 10Wikibase (3rd party installations), 06Wikibase Suite Team, 07Product-Feature: integraality for Wikibases? - https://phabricator.wikimedia.org/T294892#10779938 (10Addshore) >>! In T294892#10779885, @Olea wrote: >>>! In T294892#10779801, @Addshore wrote: > >> Everyt... [12:05:17] 10VPS-project-Codesearch: Remove dgraph-io / mediawiki-dgraph-skin from index - https://phabricator.wikimedia.org/T317432#10779993 (10A_smart_kitten) >>! In T317432#8558691, @Jdlrobson wrote: > New problem: > https://codesearch.wmcloud.org/search/?q=DgraphTemplate&i=nope&files=&excludeFiles=&repos= is not pickin... [12:34:29] 10Tool-inteGraality, 10VerySmallGLAM, 10Wikibase (3rd party installations), 06Wikibase Suite Team, 07Product-Feature: integraality for Wikibases? - https://phabricator.wikimedia.org/T294892#10780064 (10Olea) >>! In T294892#10779938, @Addshore wrote: > So these are just mediawiki / extension settings htt... [12:45:42] (03PS3) 10Btullis: Revert "Add a dummy password for rsyncing mediawiki-dumps-legacy" [labs/private] - 10https://gerrit.wikimedia.org/r/1135472 [12:47:06] (03CR) 10Btullis: [V:03+2 C:03+2] Revert "Add a dummy password for rsyncing mediawiki-dumps-legacy" [labs/private] - 10https://gerrit.wikimedia.org/r/1135472 (owner: 10Btullis) [12:47:28] (03Abandoned) 10Btullis: Revert "Add a dummy password for rsyncing mediawiki-dumps-legacy" [labs/private] - 10https://gerrit.wikimedia.org/r/1135472 (owner: 10Btullis) [12:47:58] 10Tool-inteGraality, 10VerySmallGLAM, 10Wikibase (3rd party installations), 06Wikibase Suite Team, 07Product-Feature: integraality for Wikibases? - https://phabricator.wikimedia.org/T294892#10780090 (10Addshore) Crying in public is fine :D Such configuration could also be implemented via some page on wik... [12:50:05] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: bootstrap Toolforge IaC automation - https://phabricator.wikimedia.org/T390057#10780094 (10Chuckonwumelu) 05In progress→03Resolved This is the repository for the project: https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning And... [12:55:54] 10Tool-inteGraality, 10VerySmallGLAM, 10Wikibase (3rd party installations), 06Wikibase Suite Team, 07Product-Feature: integraality for Wikibases? - https://phabricator.wikimedia.org/T294892#10780136 (10Olea) >>! In T294892#10780090, @Addshore wrote: > (but perhaps somehting we could also look at at the... [13:47:24] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision Request - Where should the new tool operation UI be available? - https://phabricator.wikimedia.org/T393010 (10Sarai-WMF) 03NEW [13:50:49] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision Request - Where should the new tool operation UI be available? - https://phabricator.wikimedia.org/T393010#10780398 (10Sarai-WMF) [14:01:31] 10Cloud-VPS (Project-requests): Request creation of osmviews VPS project with 20 GiB object storage quota - https://phabricator.wikimedia.org/T393013 (10Sascha) 03NEW [14:04:00] !log taavi@cloudcumin1001 bastion START - Cookbook wmcs.vps.add_user_to_project for user 'andreadenina' in role 'reader' [14:05:03] !log taavi@cloudcumin1001 bastion END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'andreadenina' in role 'reader' [14:13:34] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: [DRAFT] Decision Request - Where should the new tool operation UI be available? - https://phabricator.wikimedia.org/T393010#10780508 (10Sarai-WMF) [14:22:14] 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-codfw, 06SRE: decommission cloudlb2001-dev.codfw.wmnet - https://phabricator.wikimedia.org/T392686#10780556 (10Jhancock.wm) [14:24:38] 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-codfw, 06SRE: decommission cloudlb2001-dev.codfw.wmnet - https://phabricator.wikimedia.org/T392686#10780574 (10Jhancock.wm) @Andrew i can't use the offline script in netbox. looks like some of the interfaces are a little too complicated for... [14:37:06] 06cloud-services-team: Temperature Inlet Temp issue on clouddumps1001:9290 - https://phabricator.wikimedia.org/T392996#10780654 (10fnegri) →14Duplicate dup:03T383723 [14:37:09] 10cloud-services-team (FY2024/2025-Q3-Q4), 06DC-Ops, 10ops-eqiad, 06SRE: Temperature Inlet Temp issue on clouddumps1001:9290 - https://phabricator.wikimedia.org/T383723#10780656 (10fnegri) [15:40:28] FIRING: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [15:46:13] 10Cloud-Services: All tools using tools-static.wmflabs.org broken - https://phabricator.wikimedia.org/T393024 (10GPSLeo) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it with a more specific p... [15:50:18] 06cloud-services-team, 10Toolforge: All tools using tools-static.wmflabs.org broken - https://phabricator.wikimedia.org/T393024#10780950 (10JJMC89) [15:50:54] 06cloud-services-team, 10Toolforge: All tools using tools-static.wmflabs.org broken - https://phabricator.wikimedia.org/T393024#10780953 (10taavi) [15:51:03] 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: Enable IPv6 for the Cloud VPS web proxy - https://phabricator.wikimedia.org/T379175#10780954 (10taavi) [15:51:06] 06cloud-services-team, 10Cloud-VPS: lua entry thread aborted: runtime error: /etc/nginx/lua/domainproxy.lua:32: bad request - https://phabricator.wikimedia.org/T393024#10780955 (10taavi) [15:54:36] 06cloud-services-team, 10Cloud-VPS: lua entry thread aborted: runtime error: /etc/nginx/lua/domainproxy.lua:32: bad request - https://phabricator.wikimedia.org/T393024#10780965 (10taavi) p:05Triage→03High Thanks for the report! This seems to be a problem with the new proxies that {T379175} is introducing.... [16:00:28] RESOLVED: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [16:41:16] 10VPS-project-Codesearch: Remove dgraph-io / mediawiki-dgraph-skin from index - https://phabricator.wikimedia.org/T317432#10781250 (10Dzahn) The real fix is to stop using github.com and pick between Gerrit and GitLab as the official and supported places where Wikimedia hosts code. The "wikimedia organization" o... [16:42:53] 10VPS-project-Wikistats: Add rkiwiki to wikistats - https://phabricator.wikimedia.org/T392504#10781274 (10Dzahn) waiting for T392490 [16:56:04] 06Toolforge-standards-committee, 06WMF-NDA-Requests: Volunteer NDA for JJMC89 - https://phabricator.wikimedia.org/T375041#10781331 (10KFrancis) Hi all, confirming the NDA is complete. Thanks! [17:17:23] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: [DRAFT] Decision Request - Where should the new tool operation UI be available? - https://phabricator.wikimedia.org/T393010#10781444 (10Sarai-WMF) [17:35:28] FIRING: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [17:43:52] 10VPS-project-Codesearch: Remove dgraph-io / mediawiki-dgraph-skin from index - https://phabricator.wikimedia.org/T317432#10781529 (10Aklapper) > "automatically index repositories that are canonically hosted in #wikimedia-github Not possible, see T321402#8546665. [18:02:28] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance metricsinfra-prometheus-3 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [18:07:28] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance extdist-06 in project extdist - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [18:07:28] FIRING: [2x] PuppetAgentFailure: Puppet agent failure detected on instance metricsinfra-controller-2 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [18:12:28] FIRING: [4x] PuppetAgentFailure: Puppet agent failure detected on instance metricsinfra-alertmanager-2 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [18:17:28] FIRING: [5x] PuppetAgentFailure: Puppet agent failure detected on instance metricsinfra-alertmanager-2 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [18:22:28] FIRING: [6x] PuppetAgentFailure: Puppet agent failure detected on instance metricsinfra-alertmanager-2 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [18:42:48] FIRING: PuppetFailure: Puppet has failed on cloudcumin1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [18:42:53] 06cloud-services-team: PuppetFailure Puppet has failed on cloudcumin1001:9100 - https://phabricator.wikimedia.org/T393047 (10phaultfinder) 03NEW [19:02:26] (03update) 10chuckonwumelu: Importing tools dns resources [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/13 (https://phabricator.wikimedia.org/T390056) [19:03:53] (03update) 10chuckonwumelu: Importing tools dns resources [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/13 (https://phabricator.wikimedia.org/T390056) [20:43:25] (03update) 10chuckonwumelu: Importing tools dns resources [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/13 (https://phabricator.wikimedia.org/T390056) [20:52:28] RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance extdist-06 in project extdist - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [21:06:58] RESOLVED: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [21:07:58] RESOLVED: [3x] PuppetAgentFailure: Puppet agent failure detected on instance metricsinfra-alertmanager-2 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [21:10:18] RESOLVED: PuppetFailure: Puppet has failed on cloudcumin1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [21:26:06] (03PS1) 10Raymond Ndibe: [wmcs-cookbooks] run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) [21:30:17] (03PS2) 10Raymond Ndibe: [wmcs-cookbooks] run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) [21:33:00] (03update) 10raymond-ndibe: [toolforge-deploy] run specific tests on deploy [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/755 (https://phabricator.wikimedia.org/T381011) [21:33:36] (03update) 10raymond-ndibe: [toolforge-deploy] add test tags to component charts [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/755 (https://phabricator.wikimedia.org/T381011) [21:33:54] (03update) 10raymond-ndibe: [toolforge-deploy] add test tags to component charts [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/755 (https://phabricator.wikimedia.org/T381011) [21:37:47] (03CR) 10CI reject: [V:04-1] [wmcs-cookbooks] run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) (owner: 10Raymond Ndibe) [22:57:54] (03update) 10chuckonwumelu: Importing tools dns resources [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/13 (https://phabricator.wikimedia.org/T390056) [23:06:29] (03update) 10chuckonwumelu: Importing tools dns resources [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/13 (https://phabricator.wikimedia.org/T390056) [23:11:06] (03PS3) 10Raymond Ndibe: [wmcs-cookbooks] run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) [23:34:47] (03update) 10chuckonwumelu: Importing tools dns resources [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/13 (https://phabricator.wikimedia.org/T390056) [23:46:25] (03update) 10chuckonwumelu: Importing tools dns resources [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/13 (https://phabricator.wikimedia.org/T390056) [23:51:59] (03update) 10chuckonwumelu: Importing tools dns resources [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/13 (https://phabricator.wikimedia.org/T390056)