[00:15:29] FIRING: InstanceDown: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:19:13] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "image-suggestion-api" project Buster deprecation - https://phabricator.wikimedia.org/T367533#9997317 (10Andrew) 05Open→03Resolved a:03Andrew deleted. [00:20:29] RESOLVED: InstanceDown: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [01:04:35] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [01:04:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [01:04:47] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [01:04:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [01:35:02] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) [01:35:04] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [01:35:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [01:35:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [01:42:10] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [01:47:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [03:16:56] FIRING: SystemdUnitDown: The service unit opentofu-infra-diff.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [03:19:56] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:01:56] FIRING: [2x] SystemdUnitDown: The service unit purge_vm_rbd_images.service is in failed status on host cloudcontrol1005. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [04:02:03] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [04:02:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [04:02:15] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [04:02:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [04:32:14] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) [04:32:16] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [04:32:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [04:32:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [05:10:56] FIRING: SystemdUnitDown: The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [05:11:01] 06cloud-services-team: SystemdUnitDown Unit opentofu-infra-diff.service on node cloudcontrol1007 has been down for long. - https://phabricator.wikimedia.org/T370498 (10phaultfinder) 03NEW [06:58:01] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [06:58:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [06:58:13] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [06:58:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [07:19:56] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:27:32] (03update) 10dcaro: metrics: initialize all the stats to 0 [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/109 [07:28:33] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) [07:28:35] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [07:28:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [07:28:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [07:35:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [07:36:02] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [07:40:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [07:42:58] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29635 bytes in 3.573 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [07:43:01] (03update) 10dcaro: metrics: initialize all the stats to 0 [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/109 [07:50:20] (03open) 10sstefanova: d/changelog: bump to 0.0.17 [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/82 (https://phabricator.wikimedia.org/T365014) [07:50:42] (03approved) 10sstefanova: d/changelog: bump to 0.0.17 [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/82 (https://phabricator.wikimedia.org/T365014) [07:50:47] (03merge) 10sstefanova: d/changelog: bump to 0.0.17 [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/82 (https://phabricator.wikimedia.org/T365014) [08:00:32] 10Toolforge, 07Kubernetes: toolforge-jobs and packbuild images - https://phabricator.wikimedia.org/T369786#9997629 (10dcaro) Trying locally fails with: ` local.tf-test@lima-kilo:~$ toolforge jobs run --image tool-milhistbot/autocheck:latest --mount=all --command "web $OPTS" autocheck3 local.tf-test@lima-kilo:... [08:22:53] 06cloud-services-team, 10Toolforge (Toolforge iteration 13): toolforge: kubernetes can't revoke certificates - https://phabricator.wikimedia.org/T365681#9997667 (10aborrero) 05Open→03Resolved >>! In T365681#9972877, @aborrero wrote: > > The first certificates with 10 days lifetime were generated on 20... [08:46:54] 06cloud-services-team, 10Cloud-VPS, 07Epic: tofu-infra: create a cookbook automation to run tofu - https://phabricator.wikimedia.org/T370414#9997688 (10aborrero) 05Open→03In progress p:05Triage→03Medium [08:49:13] (03open) 10sstefanova: api: remove deprecated endpoints [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/106 (https://phabricator.wikimedia.org/T365014) [08:59:28] 06cloud-services-team, 10Cloud-VPS, 07Epic: tofu-infra: create a cookbook automation to run tofu - https://phabricator.wikimedia.org/T370414#9997706 (10aborrero) >>! In T370414#9994549, @fnegri wrote: > We should also make use of Spicerack's locking functionality to prevent two people from running apply at t... [09:06:56] 06cloud-services-team, 10Cloud-VPS, 07Epic: tofu-infra: create a cookbook automation to run tofu - https://phabricator.wikimedia.org/T370414#9997711 (10aborrero) I'm thinking on the following command line options and semantics: * `cookbook wmcs.openstack.tofu` <-- a no argument execution runs the plan in th... [09:11:11] FIRING: SystemdUnitDown: The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [09:16:48] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Replace deployment-ircd02 with a Bullseye or Bookworm host - https://phabricator.wikimedia.org/T369919#9997739 (10Southparkfan) 05Open→03In progress After merging https://gerrit.wikimedia.... [09:20:44] 06cloud-services-team: SystemdUnitDown Unit opentofu-infra-diff.service on node cloudcontrol1007 has been down for long. - https://phabricator.wikimedia.org/T370498#9997748 (10aborrero) I got this: `lang=shell-session aborrero@cloudcontrol1007:/srv/tofu-infra $ sudo tofu apply ╷ │ Error: error loading state: o... [09:21:55] 06cloud-services-team, 10Cloud-VPS, 07Epic: tofu-infra: create a cookbook automation to run tofu - https://phabricator.wikimedia.org/T370414#9997774 (10fnegri) > Do we make use of the mentioned etcd backend from cloudcumin hosts? I was asking @volans right at this moment :) The answer is that Spicerack lock... [09:22:25] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Rebuild or delete deployment-docker-changeprop01 - https://phabricator.wikimedia.org/T369913#9997765 (10Southparkfan) 05Open→03Invalid Instance does not exist anymore? [09:22:28] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Rebuild or delete deployment-docker-cpjobqueue01 - https://phabricator.wikimedia.org/T369914#9997772 (10Southparkfan) Instance is offline, seems to be superseded by `deployment-changeprop-1.deployment-prep.eqiad1.... [09:41:53] 10Toolforge (Toolforge iteration 13), 13Patch-For-Review: [jobs-api, jobs-cli] Prefix all endpoints with `/tool/` - https://phabricator.wikimedia.org/T363346#9997853 (10Slst2020) 05Stalled→03In progress [09:44:06] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Wikidata, 10wmde-wikidata-tech: Replace or remove Debian Buster VMs in 'wikidata-dev' cloud-vps project - https://phabricator.wikimedia.org/T360713#9997885 (10Lucas_Werkmeister_WMDE) [09:45:11] 06cloud-services-team: SystemdUnitDown Unit opentofu-infra-diff.service on node cloudcontrol1007 has been down for long. - https://phabricator.wikimedia.org/T370498#9997890 (10fnegri) 05Open→03Resolved a:03fnegri Seems to be working now: ` fnegri@cloudcontrol1007:/srv/tofu-infra$ sudo tofu plan [...... [09:46:56] RESOLVED: SystemdUnitDown: The service unit opentofu-infra-diff.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [09:49:47] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Wikidata, 10wmde-wikidata-tech: Replace or remove Debian Buster VMs in 'wikidata-dev' cloud-vps project - https://phabricator.wikimedia.org/T360713#9997895 (10Lucas_Werkmeister_WMDE) I’ve just created `wikidata-icinga-2024` as a Bookworm rep... [09:50:14] 06cloud-services-team, 10Toolforge: toolforge: upgrade all Kubernetes components to versions supporting Kubernetes 1.26 - https://phabricator.wikimedia.org/T370046#9997914 (10Slst2020) [09:51:26] RESOLVED: SystemdUnitDown: The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [09:56:35] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [09:56:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:56:47] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [09:56:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:02:42] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry [10:02:43] !log sstefanova@cloudcumin1001 tools Updating container image docker-registry.tools.wmflabs.org/nginx-ingress-controller:v1.9.6 [10:03:06] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) [10:11:29] (03open) 10sstefanova: ingress-nginx: bump version [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/433 (https://phabricator.wikimedia.org/T370046) [10:19:27] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: toolforge: upgrade all Kubernetes components to versions supporting Kubernetes 1.26 - https://phabricator.wikimedia.org/T370046#9997986 (10Slst2020) [10:20:50] (03update) 10rvogel: Add wrapper script for manual execution [toolforge-repos/cr-grants-team-metasync] - 10https://gitlab.wikimedia.org/toolforge-repos/cr-grants-team-metasync/-/merge_requests/1 [10:26:55] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) [10:26:57] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [10:26:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:27:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:27:16] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: toolforge: upgrade all Kubernetes components to versions supporting Kubernetes 1.26 - https://phabricator.wikimedia.org/T370046#9997998 (10Slst2020) [10:30:55] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: toolforge: upgrade all Kubernetes components to versions supporting Kubernetes 1.26 - https://phabricator.wikimedia.org/T370046#9998000 (10Slst2020) a:03Slst2020 Links for tekton upgrade: https://github.com/tektoncd/pipeline#migrating https://github.... [10:31:04] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Wikidata, 10wmde-wikidata-tech: Replace or remove Debian Buster VMs in 'wikidata-dev' cloud-vps project - https://phabricator.wikimedia.org/T360713#9998009 (10WMDE-leszek) Thanks for pushing @Andrew. All outdated instances are new deleted (e... [10:31:06] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Replace or delete deployment-mediawiki[11-12].deployement-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T361387#9998010 (10Southparkfan) a:03Southparkfan [10:44:04] 10Cloud-VPS (Quota-requests), 10Beta-Cluster-Infrastructure: Increase RAM and volume quotas for deployment-prep - https://phabricator.wikimedia.org/T370510 (10Southparkfan) 03NEW [10:46:18] (03PS1) 10GergesShamon: use date() instead of strftime() strftime() has been DEPRECATED as of PHP 8.1.0. Relying on this function is highly discouraged. [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1055408 (https://phabricator.wikimedia.org/T331468) [10:47:03] (03CR) 10CI reject: [V:04-1] use date() instead of strftime() strftime() has been DEPRECATED as of PHP 8.1.0. Relying on this function is highly discouraged. [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1055408 (https://phabricator.wikimedia.org/T331468) (owner: 10GergesShamon) [10:56:32] 10Cloud-VPS (Quota-requests), 10Beta-Cluster-Infrastructure: Increase RAM and volume quotas for deployment-prep - https://phabricator.wikimedia.org/T370510#9998092 (10Slst2020) +1 [11:09:50] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Remove or replace deployment-jobrunner04.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370487#9998126 (10Southparkfan) `deployment-jobrunner04` has been shut down. As... [11:16:00] FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [11:19:56] FIRING: CloudVPSDesignateLeaks: Detected 26 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:36:12] (03PS1) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [11:37:54] 06cloud-services-team, 10Cloud-VPS, 07Epic, 13Patch-For-Review: tofu-infra: create a cookbook automation to run tofu - https://phabricator.wikimedia.org/T370414#9998216 (10aborrero) I just noticed this: ` ± tofu apply --help [..] -lock=false Don't hold a state lock during the operation. This... [11:39:11] (03CR) 10CI reject: [V:04-1] cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) (owner: 10Arturo Borrero Gonzalez) [11:42:39] (03PS2) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [11:45:35] (03CR) 10CI reject: [V:04-1] cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) (owner: 10Arturo Borrero Gonzalez) [11:47:18] (03PS3) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [11:50:06] (03CR) 10CI reject: [V:04-1] cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) (owner: 10Arturo Borrero Gonzalez) [12:00:17] 06cloud-services-team, 10Cloud-VPS, 07Epic, 13Patch-For-Review: tofu-infra: create a cookbook automation to run tofu - https://phabricator.wikimedia.org/T370414#9998246 (10fnegri) > It seems tofu itself is protected against this. We may not need any lock in the cookbooks after all. Yes Tofu has a locking... [12:11:54] (03PS2) 10GergesShamon: use date() instead of strftime() strftime() has been DEPRECATED as of PHP 8.1.0. Relying on this function is highly discouraged. [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1055408 (https://phabricator.wikimedia.org/T331468) [12:30:22] (03PS4) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [12:33:52] (03CR) 10CI reject: [V:04-1] cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) (owner: 10Arturo Borrero Gonzalez) [12:42:45] (03PS5) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [12:45:26] (03CR) 10CI reject: [V:04-1] cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) (owner: 10Arturo Borrero Gonzalez) [12:46:05] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry [12:46:06] !log sstefanova@cloudcumin1001 tools Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.9.2 [12:46:17] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) [12:47:38] (03PS6) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [12:50:43] (03CR) 10CI reject: [V:04-1] cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) (owner: 10Arturo Borrero Gonzalez) [12:55:16] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [12:55:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [12:55:28] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [12:55:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [12:57:24] (03open) 10sstefanova: wmcs-k8s-metrics: bump kube-state-metrics version [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/434 (https://phabricator.wikimedia.org/T370046) [12:59:01] (03PS7) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [13:00:13] (03approved) 10aborrero: wmcs-k8s-metrics: bump kube-state-metrics version [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/434 (https://phabricator.wikimedia.org/T370046) (owner: 10sstefanova) [13:10:41] (03update) 10sstefanova: ingress-nginx: bump version [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/433 (https://phabricator.wikimedia.org/T370046) [13:21:00] FIRING: [2x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [13:26:18] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) [13:26:20] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [13:26:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [13:26:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [13:33:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [13:38:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [13:43:01] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [13:43:13] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [14:11:00] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [14:12:38] (03update) 10sstefanova: ingress-nginx: bump version [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/433 (https://phabricator.wikimedia.org/T370046) [14:16:30] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Wikidata, 10wmde-wikidata-tech: Replace or remove Debian Buster VMs in 'wikidata-dev' cloud-vps project - https://phabricator.wikimedia.org/T360713#9998580 (10Andrew) 05Open→03Resolved a:03Andrew this is great -- thanks for doing t... [14:23:01] 10Cloud-VPS (Quota-requests), 10Beta-Cluster-Infrastructure: Increase RAM and volume quotas for deployment-prep - https://phabricator.wikimedia.org/T370510#9998617 (10Andrew) 05Open→03Resolved a:03Andrew Done. Thank you so much for working on this project, @Southparkfan [14:27:56] 10Cloud-VPS: Redirect discuss-space.wmflabs.org to wmds-archive.toolforge.org - https://phabricator.wikimedia.org/T370529 (10Tgr) 03NEW [14:29:03] (03PS8) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [14:29:41] RESOLVED: CloudVPSDesignateLeaks: Detected 26 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:36:37] (03open) 10sstefanova: build: upgrade k8s dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/14 (https://phabricator.wikimedia.org/T370046) [14:56:25] (03open) 10sstefanova: build: upgrade k8s dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/10 (https://phabricator.wikimedia.org/T370046) [14:58:26] (03PS13) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [14:59:37] (03open) 10sstefanova: build: upgrade k8s dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/7 (https://phabricator.wikimedia.org/T370046) [15:00:00] (03PS14) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [15:01:56] 06Toolforge-standards-committee, 10Tools, 07Privacy: api-docs.toolforge.org loads loads 3rd party content - https://phabricator.wikimedia.org/T370532 (10bd808) 03NEW [15:03:05] (03CR) 10CI reject: [V:04-1] cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) (owner: 10Arturo Borrero Gonzalez) [15:03:10] (03PS15) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [15:05:19] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: toolforge: upgrade all Kubernetes components to versions supporting Kubernetes 1.26 - https://phabricator.wikimedia.org/T370046#9998771 (10Slst2020) [15:06:27] (03CR) 10CI reject: [V:04-1] cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) (owner: 10Arturo Borrero Gonzalez) [15:14:38] (03PS16) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [15:16:09] (03PS17) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [15:17:39] (03PS18) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [15:18:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:20:34] (03CR) 10CI reject: [V:04-1] cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) (owner: 10Arturo Borrero Gonzalez) [15:22:20] 06cloud-services-team, 10Toolforge: [infra,k8s] review kubelet flags before 1.26 upgrade - https://phabricator.wikimedia.org/T370245#9998808 (10Slst2020) [15:22:33] (03PS19) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [15:24:24] (03PS20) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [15:24:36] 06cloud-services-team, 10Toolforge: [infra,k8s] review kubelet flags before 1.26 upgrade - https://phabricator.wikimedia.org/T370245#9998815 (10Slst2020) [15:31:07] (03PS21) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [15:34:43] (03PS22) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [15:36:34] (03PS23) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [15:44:30] (03PS24) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [15:45:14] (03PS25) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [15:46:16] (03PS26) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [15:51:30] (03PS27) 10Arturo Borrero Gonzalez: cookbooks: add new wmcs.openstack.tofu cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1055420 (https://phabricator.wikimedia.org/T370414) [15:53:41] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [15:53:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:53:53] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [15:53:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:59:00] 10Data-Services, 06Security-Team, 07Security, 07Vuln-Infoleak: credentials for s56018 are compromised - https://phabricator.wikimedia.org/T370361#9998932 (10sbassett) p:05Triage→03High [16:00:23] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [16:12:43] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [16:25:08] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) [16:25:10] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [16:25:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:25:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:33:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:56:56] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Migrate deployment-prep away from Debian Buster to Bullseye/Bookworm - https://phabricator.wikimedia.org/T327742#9999065 (10Andrew) [17:26:35] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/15 (owner: 10l10n-bot) [17:26:53] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/15 (owner: 10l10n-bot) [17:31:21] (03update) 10bd808: Make image useful for Brad [toolforge-repos/bd808-buildpack-perl-bastion] - 10https://gitlab.wikimedia.org/toolforge-repos/bd808-buildpack-perl-bastion/-/merge_requests/1 [18:11:15] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [18:51:46] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [18:51:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [18:51:58] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [18:52:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [19:21:37] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) [19:21:39] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [19:21:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [19:21:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [19:49:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:59:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:30:40] 10Tool-spacemedia, 10Toolforge (Toolforge iteration 13), 10video2commons: Enable SonarCloud usage for GitHub Toolforge projects - https://phabricator.wikimedia.org/T369267#9999579 (10Don-vip) Thank you @dcaro! I managed to perform an analysis on spacemedia: - https://gitlab.wikimedia.org/toolforge-repos/spac... [20:40:25] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: Migrate eqiad1 hypervisors to Neutron OVS agent - https://phabricator.wikimedia.org/T364457#9999589 (10Andrew) [20:48:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:03:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:25:02] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Puppet-Infrastructure, 13Patch-For-Review: Ownership confusion on cloud-local puppet servers - https://phabricator.wikimedia.org/T364492#9999631 (10Andrew) I think some of what is happening here is that the python script git-sync-upstream does 'seteuid' but then s... [21:32:59] !log andrew@cloudcumin1001 cloudvirt-canary START - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary on eqiad1, with recreate True, for hosts list: ['cloudvirt1061'] [21:33:22] !log andrew@cloudcumin1001 cloudvirt-canary END (PASS) - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary (exit_code=0) on eqiad1, with recreate True, for hosts list: ['cloudvirt1061'] [21:43:53] (03open) 10bd808: Make things work on gitlab [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/1 (https://phabricator.wikimedia.org/T344744) [21:46:00] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [21:51:40] (03update) 10bd808: Make things work on gitlab [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/1 (https://phabricator.wikimedia.org/T344744) [21:51:43] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [21:51:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [21:51:55] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.depool_and_destroy [21:51:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [21:56:42] (03update) 10bd808: Make things work on gitlab [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/1 (https://phabricator.wikimedia.org/T344744) [22:01:25] (03update) 10bd808: Make things work on gitlab [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/1 (https://phabricator.wikimedia.org/T344744) [22:22:14] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) [22:22:15] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [22:22:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [22:22:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [22:25:05] (03open) 10bd808: Make things work on gitlab [toolforge-repos/ircservserv] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/merge_requests/1 (https://phabricator.wikimedia.org/T344744) [22:41:00] FIRING: [2x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [22:55:17] (03update) 10bd808: Make things work on gitlab [toolforge-repos/ircservserv] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/merge_requests/1 (https://phabricator.wikimedia.org/T344744) [23:03:13] (03update) 10bd808: Make things work on gitlab [toolforge-repos/ircservserv] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/merge_requests/1 (https://phabricator.wikimedia.org/T344744) [23:05:18] (03update) 10bd808: Make things work on gitlab [toolforge-repos/ircservserv] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/merge_requests/1 (https://phabricator.wikimedia.org/T344744) [23:11:26] (03update) 10bd808: Make things work on gitlab [toolforge-repos/ircservserv] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/merge_requests/1 (https://phabricator.wikimedia.org/T344744) [23:17:22] (03update) 10bd808: Make things work on gitlab [toolforge-repos/ircservserv] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/merge_requests/1 (https://phabricator.wikimedia.org/T344744) [23:27:44] (03update) 10bd808: Make things work on gitlab [toolforge-repos/ircservserv] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/merge_requests/1 (https://phabricator.wikimedia.org/T344744) [23:30:12] (03update) 10bd808: Make things work on gitlab [toolforge-repos/ircservserv] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/merge_requests/1 (https://phabricator.wikimedia.org/T344744) [23:35:32] (03update) 10bd808: Make things work on gitlab [toolforge-repos/ircservserv] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/merge_requests/1 (https://phabricator.wikimedia.org/T344744) [23:55:09] (03update) 10bd808: Make things work on gitlab [toolforge-repos/ircservserv] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/merge_requests/1 (https://phabricator.wikimedia.org/T344744)