[00:01:12] <wikibugs>	 (03open) 10bd808: dev: Upgrade mwclient to 0.11.0 [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/14 (https://phabricator.wikimedia.org/T372312)
[00:15:52] <wikibugs>	 (03merge) 10bd808: dev: Upgrade mwclient to 0.11.0 [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/14 (https://phabricator.wikimedia.org/T372312)
[00:20:31] <wikibugs>	 (03open) 10bd808: window: Remove debug output [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/15
[00:21:22] <wikibugs>	 (03merge) 10bd808: window: Remove debug output [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/15
[01:21:31] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[02:13:48] <wikibugs>	 10Tool-schedule-deployment: Upgrade to mwclient 0.11.0 - https://phabricator.wikimedia.org/T372312#10266189 (10bd808) 05Open→03Resolved a:03bd808
[02:31:51] <wikibugs>	 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T378311 (10IntentionallyDense) 03NEW
[02:44:51] <wikibugs>	 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T378312 (10Elisfkc) 03NEW
[02:48:11] <wikibugs>	 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T378312#10266220 (10Elisfkc) p:05Triage→03Unbreak!
[02:48:21] <wikibugs>	 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T378312#10266221 (10JJMC89) p:05Unbreak!→03Triage
[02:49:50] <wikibugs>	 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T378312#10266223 (10JJMC89) →14Duplicate dup:03T378311
[02:50:27] <wikibugs>	 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T378311#10266225 (10JJMC89)
[03:18:59] <jinxer-wm>	 FIRING: SystemdUnitDown: The service unit opentofu-infra-diff.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[03:19:45] <wikibugs>	 (03CR) 10Krinkle: "I'd prefer we do it without the nonces and instead disallow any network connections. I'm not concerned with risk of visual defacing via a " [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1080836 (https://phabricator.wikimedia.org/T377168) (owner: 10Brian Wolff)
[04:10:35] <wikibugs>	 (03open) 10bd808: build_backport_item: Escape `|` in subject [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/16 (https://phabricator.wikimedia.org/T372750)
[04:11:25] <wikibugs>	 (03merge) 10bd808: build_backport_item: Escape `|` in subject [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/16 (https://phabricator.wikimedia.org/T372750)
[04:18:16] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 10Striker, 10Tool-gitlab-account-approval, and 4 others: Removal of writeapi from siteinfo output breaks all mwclient-based bots, including stashbot (Server Admin Log) - https://phabricator.wikimedia.org/T371977#10266354 (10bd808)
[04:19:51] <wikibugs>	 10Tool-schedule-deployment, 13Patch-For-Review: schedule-deployment should escape `|` characters in values included in template parameters - https://phabricator.wikimedia.org/T372750#10266351 (10bd808) 05Open→03Resolved a:03bd808
[05:13:59] <jinxer-wm>	 FIRING: SystemdUnitDown: The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[05:21:31] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[06:09:38] <wikibugs>	 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T378311#10266441 (10Mdann52) I suspect this is related to T378320
[06:39:44] <wikibugs>	 06cloud-services-team, 10Data-Services: Give the wmcs.wikireplicas.add_wiki cookbook a way to exclude a host - https://phabricator.wikimedia.org/T273199#10266472 (10taavi)
[06:42:56] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: ProjectProxyMainProxyDown should have a runbook page - https://phabricator.wikimedia.org/T361873#10266473 (10taavi)
[08:31:01] <wikibugs>	 (03PS1) 10Stevemunene: Add dummy keytabs for new presto hosts [labs/private] - 10https://gerrit.wikimedia.org/r/1083755 (https://phabricator.wikimedia.org/T374924)
[08:33:08] <wikibugs>	 (03PS2) 10Stevemunene: Add dummy keytabs for new presto hosts [labs/private] - 10https://gerrit.wikimedia.org/r/1083755 (https://phabricator.wikimedia.org/T374924)
[09:00:43] <wikibugs>	 (03update) 10raymond-ndibe: [maintain-harbor] Move to become a toolforge component [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/34 (https://phabricator.wikimedia.org/T358225)
[09:12:55] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867#10266726 (10dcaro)
[09:12:59] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867#10266727 (10dcaro)
[09:13:59] <jinxer-wm>	 FIRING: SystemdUnitDown: The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[09:21:31] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[09:27:45] <wikibugs>	 06cloud-services-team, 10Toolforge: Add --timeout to toolforge jobs - https://phabricator.wikimedia.org/T377782#10266771 (10dcaro) As a workaround for now, this can be done by using the `timeout` command wrapping your job command, like:  ` toolforge jobs run --schedule '00 * * * * *' --command 'timeout 1h myco...
[09:46:17] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 16): Introduce health checks for Toolforge Jobs Framework cronjobs - https://phabricator.wikimedia.org/T377420#10266826 (10dcaro) Similar case on customizing k8s concurrency {https://phabricator.wikimedia.org/T375366}  Maybe it's not such a bad idea to ad...
[09:47:44] <wikibugs>	 (03PS1) 10Slyngshede: IDM: Redis dummy password [labs/private] - 10https://gerrit.wikimedia.org/r/1083764 (https://phabricator.wikimedia.org/T377728)
[09:48:55] <wikibugs>	 (03PS2) 10Slyngshede: IDM: Redis dummy password [labs/private] - 10https://gerrit.wikimedia.org/r/1083764 (https://phabricator.wikimedia.org/T377728)
[09:54:36] <wikibugs>	 (03PS3) 10Slyngshede: IDP: Redis dummy password [labs/private] - 10https://gerrit.wikimedia.org/r/1083764 (https://phabricator.wikimedia.org/T377728)
[10:26:12] <wikibugs>	 (03CR) 10Slyngshede: [V:03+2 C:03+2] IDP: Redis dummy password [labs/private] - 10https://gerrit.wikimedia.org/r/1083764 (https://phabricator.wikimedia.org/T377728) (owner: 10Slyngshede)
[11:12:01] <wikibugs>	 (03CR) 10Brouberol: [C:03+1] Add dummy keytabs for new presto hosts [labs/private] - 10https://gerrit.wikimedia.org/r/1083755 (https://phabricator.wikimedia.org/T374924) (owner: 10Stevemunene)
[11:25:01] <wikibugs>	 10Cloud Services Proposals, 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Decision Request  - How to do the Cloud VPS VXLAN/IPv6 migration - https://phabricator.wikimedia.org/T377467#10267254 (10dcaro) I don't have a strong opinion, mostly questions xd  For what I understand, trying to summarize wh...
[11:50:56] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge, 05Goal: [toolsdb] Upgrade to MariaDB 10.6 - https://phabricator.wikimedia.org/T352206#10267441 (10fnegri)
[12:18:59] <jinxer-wm>	 RESOLVED: SystemdUnitDown: The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[12:20:43] <wikibugs>	 (03update) 10raymond-ndibe: [maintain-harbor] Move to become a toolforge component [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/34 (https://phabricator.wikimedia.org/T358225)
[12:20:57] <wikibugs>	 10Cloud Services Proposals, 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Decision Request  - How to do the Cloud VPS VXLAN/IPv6 migration - https://phabricator.wikimedia.org/T377467#10267530 (10aborrero) thanks for chiming in, some replies inline.  >>! In T377467#10267254, @dcaro wrote: >  >>...
[12:52:06] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_tool_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[12:57:06] <wmcs-alerts>	 RESOLVED: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[12:57:55] <wikibugs>	 10Cloud Services Proposals, 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Decision Request  - How to do the Cloud VPS VXLAN/IPv6 migration - https://phabricator.wikimedia.org/T377467#10267614 (10aborrero) >>! In T377467#10267551, @taavi wrote: >> 2025-02-01: (2 months later) option to create VMs in...
[13:21:31] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[13:36:03] <wmcs-alerts>	 FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-27 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses
[13:41:03] <wmcs-alerts>	 RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-27 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses
[14:09:50] <wikibugs>	 10Cloud Services Proposals, 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Decision Request  - How to do the Cloud VPS VXLAN/IPv6 migration - https://phabricator.wikimedia.org/T377467#10267875 (10dcaro) >>>! In T377467#10267254, @dcaro wrote: >>  >>>     how much effort we want to put into it >> Thi...
[15:08:43] <wikibugs>	 10Quarry: Set query result retention time - https://phabricator.wikimedia.org/T360041#10268152 (10rook) I appreciate the commentary. Though none of it gets at the central issue of PII, and the reality that quarry is not designed to keep data in perpetuity. Data persistence is an expensive process and not being a...
[15:14:40] <wikibugs>	 06cloud-services-team, 10Toolforge: [infra, k8s, webservice] remove deprecated kubectl --wait flag before k8s 1.29 upgrade - https://phabricator.wikimedia.org/T373866#10268158 (10bd808) 05Resolved→03Open This deployment seems to still be pending.  `lang=shell-session tools.schedule-deployment@tools-bastion...
[15:35:14] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 06Tech-Docs-Team, 07Documentation: WMCS: Document different types of root and admin privileges - https://phabricator.wikimedia.org/T375113#10268310 (10fnegri) 05In progress→03Resolved @TBurmeister thanks for the review! I followed most of your...
[15:48:15] <wikibugs>	 10Cloud Services Proposals, 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Decision Request  - How to do the Cloud VPS VXLAN/IPv6 migration - https://phabricator.wikimedia.org/T377467#10268424 (10Dzahn) As someone using cloud VPS with puppet the one major advantage for users like me would be that pu...
[16:12:06] <wikibugs>	 06cloud-services-team, 10Toolforge: Timeouts when pulling artifacts from github with toolforge build - https://phabricator.wikimedia.org/T377255#10268606 (10dcaro) @Don-vip has this happened again? Otherwise it might have been a temporary blip
[16:13:04] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: [openstack object storage] deleted files still occupying space - https://phabricator.wikimedia.org/T376673#10268620 (10dcaro)
[16:13:10] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge, 05Goal: [harbor] Move harbor data to object storage service - https://phabricator.wikimedia.org/T350687#10268621 (10dcaro)
[16:23:29] <wikibugs>	 10Tool-schedule-deployment: Internal Server Error when attempting to schedule a backport - https://phabricator.wikimedia.org/T378379 (10kostajh) 03NEW
[16:30:04] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: [openstack object storage] deleted files still occupying space - https://phabricator.wikimedia.org/T376673#10268775 (10dcaro) This might be due to the rados gw nodes (cloudcontrols) running ceph v16, while the manager nodes (cloudcephmon) run ceph v15, so...
[16:40:43] <wikibugs>	 10Quarry: Set query result retention time - https://phabricator.wikimedia.org/T360041#10268826 (10rook) 05Open→03Resolved a:03rook
[16:51:44] <icinga-wm>	 PROBLEM - ensure kvm processes are running on cloudvirt1063 is CRITICAL: PROCS CRITICAL: 0 processes with regex args qemu-system-x86_64 https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting
[16:53:43] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack
[16:53:44] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
[16:54:14] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack
[16:54:15] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
[16:54:18] <wikibugs>	 10Tool-video-answer-tool, 06Future-Audiences: Add last modified and # of contributors metadata to video attribution - https://phabricator.wikimedia.org/T378383 (10Maryana) 03NEW
[16:58:55] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: neutron: clarify why DNS extension is not enabled - https://phabricator.wikimedia.org/T377740#10268995 (10aborrero) 05In progress→03Stalled
[17:00:20] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack
[17:01:36] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
[17:21:31] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[17:35:55] <wikibugs>	 10Tool-schedule-deployment: Internal Server Error when attempting to schedule a backport - https://phabricator.wikimedia.org/T378379#10269324 (10bd808) 05Open→03In progress p:05Triage→03High a:03bd808
[17:36:03] <wikibugs>	 10Quarry: Set query result retention time - https://phabricator.wikimedia.org/T360041#10269341 (10Novem_Linguae) > Though none of it gets at the central issue of PII  I believe PII such as email addresses, password hashes, and IPs is scrubbed by the replicas? Quarry isn't a system I think of as having PII in...
[17:36:13] <wikibugs>	 10Tool-schedule-deployment: Internal Server Error when attempting to schedule a backport - https://phabricator.wikimedia.org/T378379#10269347 (10bd808) `lang=python 2024-10-28T09:41:51Z deployments.web ERROR: Exception on / [GET] Traceback (most recent call last):   File "/layers/heroku_python/dependencies/lib/p...
[17:46:34] <wikibugs>	 10Quarry: Set query result retention time - https://phabricator.wikimedia.org/T360041#10269413 (10rook) > I believe PII such as email addresses, password hashes, and IPs is scrubbed by the replicas? Quarry isn't a system I think of as having PII in it. All the data it queries is public, I think.  The issue i...
[17:48:22] <wikibugs>	 10Tool-schedule-deployment: Internal Server Error when attempting to schedule a backport - https://phabricator.wikimedia.org/T378379#10269426 (10bd808) Things are blowing up because https://wikitech.wikimedia.org/w/index.php?title=Deployments&diff=prev&oldid=2239184 introduced an unexpected timezone of "CET" and...
[17:54:57] <wikibugs>	 10Tool-schedule-deployment: 500 error when an unexpected timezone is found for a backport window - https://phabricator.wikimedia.org/T378379#10269499 (10bd808)
[17:57:17] <wikibugs>	 10Tool-schedule-deployment: 500 error when an unexpected timezone is found for a backport window - https://phabricator.wikimedia.org/T378379#10269504 (10bd808) p:05High→03Medium Applied a bandaid by making the wikitext use an expected timezone specifier: https://wikitech.wikimedia.org/w/index.php?title=Deplo...
[18:02:21] <wikibugs>	 (03CR) 10David Caro: [C:03+2] toolforge.component.deploy: handle non-mr package deploys [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1078676 (owner: 10David Caro)
[18:13:18] <wikibugs>	 (03Merged) 10jenkins-bot: toolforge.component.deploy: handle non-mr package deploys [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1078676 (owner: 10David Caro)
[19:08:33] <wikibugs>	 06cloud-services-team, 10Toolforge: Timeouts when pulling artifacts from github with toolforge build - https://phabricator.wikimedia.org/T377255#10269918 (10Don-vip) @dcaro no it happened only once for me. Feel free to close the ticket :)
[19:16:25] <wikibugs>	 06Toolforge-standards-committee: Reset members and owners for toolforge-standards-committee@lists.wikimedia.org - https://phabricator.wikimedia.org/T375134#10269932 (10bd808)
[19:20:13] <wikibugs>	 10Tool-schedule-deployment: 500 error when an unexpected timezone is found for a backport window - https://phabricator.wikimedia.org/T378379#10269958 (10kostajh) >>! In T378379#10269499, @bd808 wrote: > Applied a bandaid by making the wikitext use an expected timezone specifier: https://wikitech.wikimedia.org/w/...
[19:47:49] <wikibugs>	 10Tool-schedule-deployment: 500 error when Template:Ircnick is used with a single argument - https://phabricator.wikimedia.org/T378412 (10bd808) 03NEW
[19:48:42] <wikibugs>	 10Tool-schedule-deployment: 500 error when Template:Ircnick is used with a single argument - https://phabricator.wikimedia.org/T378412#10270143 (10bd808) 05Open→03In progress p:05Triage→03Medium a:03bd808
[20:14:41] <wikibugs>	 (03open) 10bd808: Fix real world data handling bugs [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/17 (https://phabricator.wikimedia.org/T378379 https://phabricator.wikimedia.org/T378412)
[20:16:00] <wikibugs>	 (03merge) 10bd808: Fix real world data handling bugs [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/17 (https://phabricator.wikimedia.org/T378379 https://phabricator.wikimedia.org/T378412)
[20:35:25] <wikibugs>	 10Tool-schedule-deployment: 500 error when Template:Ircnick is used with a single argument - https://phabricator.wikimedia.org/T378412#10270346 (10bd808) 05In progress→03Resolved
[20:35:38] <wikibugs>	 10Tool-schedule-deployment: 500 error when an unexpected timezone is found for a backport window - https://phabricator.wikimedia.org/T378379#10270347 (10bd808) 05In progress→03Resolved
[21:15:23] <jinxer-wm>	 FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse
[21:18:10] <wmcs-alerts>	 FIRING: PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-harbor-1 on project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[21:21:31] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[22:20:45] <wikibugs>	 10Cloud Services Proposals, 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Decision Request  - How to do the Cloud VPS VXLAN/IPv6 migration - https://phabricator.wikimedia.org/T377467#10270652 (10bd808) >>! In T377467#10267530, @aborrero wrote: > * we can rethink how we do floating IPs. If you want...
[22:47:38] <wmcs-alerts>	 FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate mwv-builder-03.mediawiki-vagrant.eqiad.wmflabs is about to expire in -3d 0h 1m 25s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire
[22:52:54] <wikibugs>	 10Cloud Services Proposals, 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Decision Request  - How to do the Cloud VPS VXLAN/IPv6 migration - https://phabricator.wikimedia.org/T377467#10270687 (10JJMC89) >>! In T377467#10270652, @bd808 wrote: > There are a small number of projects that we have allow...