[01:22:36] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[04:59:30] <wikibugs>	 (03open) 10raymond-ndibe: [maintain-harbor] specify config through environment variables [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/36 (https://phabricator.wikimedia.org/T358225)
[05:02:14] <wikibugs>	 (03update) 10raymond-ndibe: [maintain-harbor] Move to become a toolforge component [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/34 (https://phabricator.wikimedia.org/T358225)
[05:22:36] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[05:28:15] <wikibugs>	 (03update) 10raymond-ndibe: [maintain-harbor] Move to become a toolforge component [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/34 (https://phabricator.wikimedia.org/T358225)
[05:29:30] <wikibugs>	 (03update) 10raymond-ndibe: [maintain-harbor] Move to become a toolforge component [repos/cloud/toolforge/maintain-harbor] (allow_getting_config_from_env) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/34 (https://phabricator.wikimedia.org/T358225)
[05:30:47] <wikibugs>	 (03update) 10raymond-ndibe: [maintain-harbor] Move to become a toolforge component [repos/cloud/toolforge/maintain-harbor] (allow_getting_config_from_env) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/34 (https://phabricator.wikimedia.org/T358225)
[05:31:11] <wikibugs>	 (03update) 10raymond-ndibe: [maintain-harbor] Move to become a toolforge component [repos/cloud/toolforge/maintain-harbor] (allow_getting_config_from_env) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/34 (https://phabricator.wikimedia.org/T358225)
[06:33:59] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/75
[06:34:05] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/34
[06:34:06] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/33
[06:34:09] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/75
[06:34:17] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/63
[06:34:29] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/16
[06:34:31] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/34
[06:34:40] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/63
[06:34:40] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/13
[06:34:48] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/16
[06:34:57] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/92
[06:34:59] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/93
[06:35:00] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/13
[06:35:11] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/92
[06:35:16] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/93
[06:35:23] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/57
[06:35:44] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/127
[06:35:44] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/57
[06:35:45] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/65
[06:35:46] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/66
[06:35:47] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/128
[06:35:48] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/127
[06:35:59] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/128
[06:36:11] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/14
[06:36:14] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/565
[06:36:15] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/20
[06:36:15] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/46
[06:36:19] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/65
[06:36:23] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/66
[06:36:25] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/31
[06:36:29] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/47
[06:36:31] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/14
[06:36:35] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/46
[06:36:35] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/20
[06:36:40] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/114
[06:36:44] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/47
[06:36:48] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/565
[06:36:52] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/15
[06:36:56] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/62
[06:37:00] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/206
[06:37:04] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/37
[06:37:08] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/47
[06:37:12] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/31
[06:37:16] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/wmcs-k8s-metrics] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/wmcs-k8s-metrics/-/merge_requests/10
[06:37:20] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/114
[06:37:24] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/32
[06:37:28] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/47
[06:37:32] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/15
[06:37:36] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/62
[06:37:40] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/206
[06:37:44] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/37
[06:37:48] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/wmcs-k8s-metrics] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/wmcs-k8s-metrics/-/merge_requests/10
[06:37:52] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/32
[09:22:36] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 5 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[10:34:49] <wikibugs>	 06cloud-services-team, 10Citoid: Investigate Wikipedia bot/userscript usage of citoid impacting www.pro-football-reference.com - https://phabricator.wikimedia.org/T378461#10283792 (10Mvolz)
[10:36:48] <wikibugs>	 10Tool-refill: refill: review citoid usage - https://phabricator.wikimedia.org/T378686#10283791 (10Mvolz) >>! In T378686#10282814, @Curb_Safe_Charmer wrote: > There is certainly nothing in the service log that indicates that reFill was called 30,000 times in any 12 hours.   Thanks for looking into this!   We...
[10:38:29] <wmcs-alerts>	 FIRING: PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-idp-1 on project cloudinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:40:38] <wikibugs>	 10Tool-refill: refill: review citoid usage - https://phabricator.wikimedia.org/T378686#10283800 (10Mvolz) I see in the other thread you don't have Turnilo access, sorry. The user-agent is 'reFill/2 (http://en.wikipedia.org/wiki/User:Zhaofeng_Li/reFill)'   I'm not sure what the privacy policy is re:IPs but ht...
[13:22:36] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 5 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[13:51:53] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services: maintain_meta-p prints warning related to labtestwikitech - https://phabricator.wikimedia.org/T378687#10284115 (10fnegri) 05Stalled→03Resolved @taavi no, this is fixed!
[13:52:39] <wmcs-alerts>	 FIRING: [2x] PuppetCertificateAboutToExpire: Puppet CA certificate deployment-poolcounter06.deployment-prep.eqiad.wmflabs is about to expire in 26d 23h 58m 30s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire
[14:03:11] <wmcs-alerts>	 FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance toolsbeta-test-k8s-haproxy-6 in project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[14:45:18] <wm-bot2>	 !log dcaro@urcuchillay dcaro-test1 END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project dcaro-test1 in codfw1dev
[14:45:20] <stashbot>	 wmbot~dcaro@urcuchillay: Unknown project "dcaro-test1"
[14:57:17] <wikibugs>	 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: cloudcontrol2006-dev struggling with memory - https://phabricator.wikimedia.org/T370401#10284297 (10Jhancock.wm) 05In progress→03Resolved a:05Papaul→03Jhancock.wm upgraded to 128Gb. powering up now. Please let us know if you need a...
[16:14:44] <wikibugs>	 10Tools: I keep getting Toolforge messages about rustup - https://phabricator.wikimedia.org/T378437#10284448 (10taavi) /cc @Legoktm
[16:20:19] <wikibugs>	 06cloud-services-team, 10Data-Services: Create views for DiscussionTools items tables - https://phabricator.wikimedia.org/T374584#10284460 (10taavi)
[16:51:23] <wikibugs>	 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw: Q#:rack/setup/install cloudcephosd2004-dev - https://phabricator.wikimedia.org/T378825 (10RobH) 03NEW
[16:51:34] <wikibugs>	 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw: Q2:rack/setup/install cloudcephosd2004-dev - https://phabricator.wikimedia.org/T378825#10284584 (10RobH)
[16:53:31] <wikibugs>	 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw: Q2:rack/setup/install cloudcephosd2004-dev - https://phabricator.wikimedia.org/T378825#10284594 (10RobH) a:03Andrew @andrew,  Please note the workflow for racking tasks has changed this fiscal year, and we now require the puppet updates from the sub-...
[16:54:23] <wikibugs>	 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw: Q2:rack/setup/install cloudcephosd2004-dev - https://phabricator.wikimedia.org/T378825#10284607 (10RobH)
[17:22:36] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[18:04:16] <wikibugs>	 10Cloud-VPS (Quota-requests): Increase project catalyst to VCPUs +16, RAM +32GB - https://phabricator.wikimedia.org/T378841 (10SDunlap) 03NEW
[18:43:11] <jinxer-wm>	 FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[18:43:22] <wikibugs>	 06cloud-services-team: Temperature clouddumps1001:9290 Inlet Temp issue on clouddumps1001:9290 - https://phabricator.wikimedia.org/T378844 (10phaultfinder) 03NEW
[18:48:11] <jinxer-wm>	 RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature
[18:52:15] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Create mechanism to allow the use of vanity domains by projects behind the Cloud VPS shared HTTP proxy - https://phabricator.wikimedia.org/T342398#10285184 (10taavi) a:03taavi Next up I think is to find a real project with real traffic that wants to give this a try.
[18:55:57] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Migrate projects with floating IPs for web traffic via custom domains to the shared proxy - https://phabricator.wikimedia.org/T378845 (10taavi) 03NEW
[18:58:35] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 10Data-Services, 13Patch-For-Review: tofu-infra: replace wmcs-wikireplica-dns.py with tofu - https://phabricator.wikimedia.org/T374953#10285221 (10taavi)
[18:59:43] <wikibugs>	 06cloud-services-team, 10Horizon: horizon: enable IPv6 security group panels - https://phabricator.wikimedia.org/T377339#10285227 (10taavi)
[19:00:13] <wikibugs>	 06cloud-services-team, 10wikitech.wikimedia.org, 06Data-Persistence: Decommission clouddb2002-dev.codfw.wmnet - https://phabricator.wikimedia.org/T369308#10285228 (10taavi)
[19:02:16] <wikibugs>	 10Cloud-VPS (Quota-requests): Increase project catalyst to VCPUs +16, RAM +32GB - https://phabricator.wikimedia.org/T378841#10285255 (10SDunlap)
[19:03:10] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Update OpenStack nova API endpoint version - https://phabricator.wikimedia.org/T275084#10285276 (10taavi)
[19:03:44] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 10netbox, 10Puppet-Core: Make netbox the source of truth for cloudceph networks - https://phabricator.wikimedia.org/T338329#10285243 (10taavi)
[19:03:46] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: [openstack] Review HAproxy setup to achieve load balancing or at least automatic failover - https://phabricator.wikimedia.org/T269687#10285238 (10taavi) 05Open→03Resolved a:03taavi This is done with the cloudlb project.
[19:10:44] <wikibugs>	 10Tool-ldap: LDAP tool list groups that do not exist - https://phabricator.wikimedia.org/T378847 (10Bugreporter) 03NEW
[19:13:10] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 07LDAP: novaadmin LDAP user is a member of nonexistent LDAP groups - https://phabricator.wikimedia.org/T378847#10285335 (10taavi)
[19:13:29] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 07LDAP: novaadmin LDAP user is a member of nonexistent LDAP groups - https://phabricator.wikimedia.org/T378847#10285336 (10taavi) `lang=shell-session taavi@tools-bastion-12:~ $ ldapsearch -x cn=novaadmin memberOf | grep wikinewsie memberOf: cn=wikinewsie,ou=projects,dc=wik...
[19:15:38] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 07LDAP: novaadmin LDAP user is a member of nonexistent LDAP groups - https://phabricator.wikimedia.org/T378847#10285332 (10taavi) That's an issue in the underlying data in LDAP, which the tool is showing correctly enough.
[19:16:57] <wikibugs>	 10Cloud-VPS (Quota-requests): Increase project catalyst to VCPUs +16, RAM +32GB - https://phabricator.wikimedia.org/T378848 (10SDunlap) 03NEW
[19:20:13] <wikibugs>	 10VPS-project-Wikistats: wikistats: mediawiki version shown incorrectly in "largest wikis" listing - https://phabricator.wikimedia.org/T317241#10285357 (10Dzahn) https://gitlab.wikimedia.org/cloudvps-repos/wikistats/-/commit/23d14d562a84eba0501d8395cbc873442d21a317
[21:18:12] <wmcs-alerts>	 FIRING: PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-harbor-1 on project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[21:22:36] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[22:16:03] <wikibugs>	 10Tool-refill: refill: review citoid usage - https://phabricator.wikimedia.org/T378686#10285737 (10akosiaris) The IPs listed in turnilo are internal toolforge IPs. Specifically 172.16.5.177 (for reFill), which resolves to a toolforge kubernetes worker, tools-k8s-worker-nfs-11.tools.eqiad1.wikimedia.cloud. Th...
[22:17:35] <wikibugs>	 06cloud-services-team, 10Citoid: Investigate Wikipedia bot/userscript usage of citoid impacting www.pro-football-reference.com - https://phabricator.wikimedia.org/T378461#10285753 (10akosiaris) Hi @Curb_Safe_Charmer, thanks for taking a look.  >>! In T378461#10282956, @Curb_Safe_Charmer wrote: > I couldn't acc...
[22:41:09] <wikibugs>	 10VPS-project-Wikistats: wikistats: mediawiki version shown incorrectly in "largest wikis" listing - https://phabricator.wikimedia.org/T317241#10285762 (10Dzahn) 05Open→03Resolved a:03Dzahn I changed all update scripts to use the "extinfo" update method which pulls in the version and changed version to...
[22:53:31] <wikibugs>	 10Tool-refill: refill: review citoid usage - https://phabricator.wikimedia.org/T378686#10285768 (10Curb_Safe_Charmer) Do these definitely resolve to refill?
[23:34:00] <wikibugs>	 10Tool-refill: refill: review citoid usage - https://phabricator.wikimedia.org/T378686#10285777 (10JJMC89) The refill front end has been running on tools-k8s-worker-nfs-9 since 2024-10-17T20:36:11Z per https://k8s-status.toolforge.org/namespaces/tool-refill/. It could have been on a different worker before t...