[00:04:36] <wikibugs>	 (03PS1) 10Andrew Bogott: Fake passwords for trove rabbitmq user [labs/private] - 10https://gerrit.wikimedia.org/r/1058720 (https://phabricator.wikimedia.org/T320256)
[00:08:17] <wikibugs>	 (03CR) 10Andrew Bogott: [V:03+2 C:03+2] Fake passwords for trove rabbitmq user [labs/private] - 10https://gerrit.wikimedia.org/r/1058720 (https://phabricator.wikimedia.org/T320256) (owner: 10Andrew Bogott)
[00:10:50] <wmcs-alerts>	 FIRING: TfInfraTestDestroyFailed: Terraform failed to destroy the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed
[00:16:56] <jinxer-wm>	 FIRING: SystemdUnitDown: The service unit wikitech_run_jobs.service is in failed status on host cloudweb1004. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudweb1004 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[00:19:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[00:21:56] <jinxer-wm>	 FIRING: [3x] SystemdUnitDown: The service unit kiwix-mirror-update.service is in failed status on host clouddumps1002. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown  - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[00:36:56] <jinxer-wm>	 FIRING: [3x] SystemdUnitDown: The service unit kiwix-mirror-update.service is in failed status on host clouddumps1002. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown  - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[00:38:53] <wikibugs>	 (03PS3) 10XtexChooser: labsauth: Set OATH input autocomplete to one-time-code [labs/striker] - 10https://gerrit.wikimedia.org/r/1058616
[00:40:25] <wikibugs>	 (03CR) 10XtexChooser: labsauth: Set OATH input autocomplete to one-time-code (031 comment) [labs/striker] - 10https://gerrit.wikimedia.org/r/1058616 (owner: 10XtexChooser)
[01:29:33] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack
[01:30:10] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
[01:31:48] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack
[01:32:25] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
[01:40:30] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack
[01:40:41] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
[01:54:30] <wikibugs>	 10Cloud-VPS, 13Patch-For-Review: puppet problems mounting cinder volumes (and suggested fixes) - https://phabricator.wikimedia.org/T371573#10034294 (10Dzahn)
[01:56:14] <wikibugs>	 10Cloud-VPS, 13Patch-For-Review: puppet problems mounting cinder volumes (and suggested fixes) - https://phabricator.wikimedia.org/T371573#10034295 (10Dzahn)
[01:56:45] <wikibugs>	 10Cloud-VPS, 06collaboration-services, 13Patch-For-Review: puppet problems mounting cinder volumes (and suggested fixes) - https://phabricator.wikimedia.org/T371573#10034296 (10Dzahn)
[02:02:16] <wikibugs>	 10Cloud-VPS, 06collaboration-services, 13Patch-For-Review: puppet problems mounting cinder volumes (and suggested fixes) - https://phabricator.wikimedia.org/T371573#10034298 (10Dzahn) This is relevant for me in the `wikistats` projected and sre-collab for T371066. So we could let puppet mount volumes without...
[02:16:56] <jinxer-wm>	 FIRING: SystemdUnitDown: The systemd unit kiwix-mirror-update.service on node clouddumps1002 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1002 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[02:17:08] <wikibugs>	 06cloud-services-team: SystemdUnitDown  Unit kiwix-mirror-update.service on node clouddumps1002 has been down for long. - https://phabricator.wikimedia.org/T371576 (10phaultfinder) 03NEW
[02:43:17] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack
[02:43:29] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
[03:04:17] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack
[03:04:27] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
[03:04:33] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack
[03:04:43] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
[03:59:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[04:50:36] <wikibugs>	 10Data-Services, 06Data-Engineering, 06DBA: Prepare and check storage layer for cswikivoyage - https://phabricator.wikimedia.org/T370912#10034384 (10Marostegui)
[05:38:11] <wikibugs>	 10Toolforge (Toolforge iteration 13): Toolforge Aptfile not producing working copy of `ffmpeg` - https://phabricator.wikimedia.org/T365633#10034406 (10tchin) I think I'm experiencing a similar error. Suddenly started getting this on a scraping job I have: ` 2024-08-01T02:42:58+00:00 [test-scrape-pd5kq] â Host sy...
[05:53:21] <wikibugs>	 10Toolforge: Please re-install "joe" - https://phabricator.wikimedia.org/T371556#10034412 (10Plenz) Thank you very much! I changed from "login.toolforge.org" to "login-buster.toolforge.org" and joe is available again.
[05:55:14] <wikibugs>	 10Toolforge: Please re-install "joe" - https://phabricator.wikimedia.org/T371556#10034413 (10Plenz) 05Open→03Resolved a:03Plenz I hope this is a permanent solution, so I close this ticket.
[05:57:37] <wikibugs>	 (03update) 10sstefanova: bump openapi version [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/30 (https://phabricator.wikimedia.org/T356974) (owner: 10dcaro)
[06:17:11] <jinxer-wm>	 FIRING: SystemdUnitDown: The systemd unit kiwix-mirror-update.service on node clouddumps1002 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1002 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[06:33:55] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/2
[06:33:57] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/61
[06:34:02] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/61
[06:34:03] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/3
[06:34:11] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/84
[06:34:20] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/84
[06:34:28] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/50
[06:34:32] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/111
[06:34:32] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/112
[06:34:34] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/50
[06:34:48] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/55
[06:34:49] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/111
[06:34:50] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/56
[06:34:51] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/55
[06:34:52] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/112
[06:34:56] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/56
[06:35:14] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/31
[06:35:19] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/31
[06:35:22] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/43
[06:35:23] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/24
[06:35:26] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/43
[06:35:28] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/107
[06:35:28] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/107
[06:35:29] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/24
[06:35:33] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/32
[06:35:37] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/32
[06:35:41] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/180
[06:35:45] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/31
[06:35:49] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/180
[06:35:53] <wikibugs>	 (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/25
[06:35:57] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/31
[06:36:01] <wikibugs>	 (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/25
[06:36:39] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-6:30000 has failed probes (http_this_tool_does_not_exist_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#toolsbeta-test-k8s-haproxy-6:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[06:41:39] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-6:30000 has failed probes (http_this_tool_does_not_exist_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#toolsbeta-test-k8s-haproxy-6:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[06:48:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[06:58:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[08:16:56] <jinxer-wm>	 RESOLVED: SystemdUnitDown: The service unit kiwix-mirror-update.service is in failed status on host clouddumps1002. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1002 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[08:16:56] <jinxer-wm>	 RESOLVED: SystemdUnitDown: The systemd unit kiwix-mirror-update.service on node clouddumps1002 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1002 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[08:27:31] <wikibugs>	 10Data-Services, 06Data-Engineering, 06Trust and Safety Product Team: Hide the value of gb_address column in public replicas if gb_autoblock_parent_id is not null - https://phabricator.wikimedia.org/T371486#10034588 (10fnegri)
[08:28:01] <wikibugs>	 10Data-Services, 06Data-Engineering, 06Trust and Safety Product Team: Hide rows in the globalblocks table when the associated globaluser row has gu_hidden_level as not 0 - https://phabricator.wikimedia.org/T371488#10034589 (10fnegri)
[09:03:04] <wikibugs>	 10Cloud-VPS (Quota-requests), 06collaboration-services: Increase quota for volumes in devtools project - https://phabricator.wikimedia.org/T371093#10034644 (10fnegri) +1
[09:04:15] <wikibugs>	 10Cloud-VPS (Quota-requests), 10video2commons: Request: add +16 cpu / +16 Gb ram to video project quota - https://phabricator.wikimedia.org/T371047#10034662 (10fnegri) +1 but let's revert back the quotas after the migration is completed.
[09:06:16] <wikibugs>	 10Cloud-VPS (Quota-requests), 06collaboration-services: Increase quota for volumes in devtools project - https://phabricator.wikimedia.org/T371093#10034681 (10Slst2020) a:03Slst2020
[09:06:56] <wikibugs>	 10Cloud-VPS (Quota-requests), 10video2commons: Request: add +16 cpu / +16 Gb ram to video project quota - https://phabricator.wikimedia.org/T371047#10034685 (10Slst2020) a:03Slst2020
[09:12:54] <logmsgbot_cloud>	 !log sstefanova@cloudcumin1001 video START - Cookbook wmcs.openstack.quota_increase
[09:13:02] <logmsgbot_cloud>	 !log sstefanova@cloudcumin1001 video END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
[09:16:03] <wikibugs>	 10Cloud-VPS (Quota-requests), 10video2commons: Request: add +16 cpu / +16 Gb ram to video project quota - https://phabricator.wikimedia.org/T371047#10034690 (10Slst2020) 05Open→03Resolved done! please reopen this task with a comment once the migration is done and you don't need the extra quotas anymore
[09:21:25] <logmsgbot_cloud>	 !log sstefanova@cloudcumin1001 devtools START - Cookbook wmcs.openstack.quota_increase
[09:21:33] <logmsgbot_cloud>	 !log sstefanova@cloudcumin1001 devtools END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
[09:24:31] <wikibugs>	 10Cloud-VPS (Quota-requests), 06collaboration-services: Increase quota for volumes in devtools project - https://phabricator.wikimedia.org/T371093#10034695 (10Slst2020) 05Open→03Resolved Done, you have a total of 150GB now.
[09:41:10] <wikibugs>	 (03open) 10sstefanova: api: add boilerplate for endpoints [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/4 (https://phabricator.wikimedia.org/T362069)
[09:48:13] <wikibugs>	 (03open) 10sstefanova: functional-tests: add direct-api tests for components api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/464
[10:05:00] <wikibugs>	 (03update) 10sstefanova: api: add boilerplate for endpoints [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/4 (https://phabricator.wikimedia.org/T362069)
[10:13:35] <wikibugs>	 (03PS1) 10Clément Goubert: Remove obsolete dummy certs for api and parsoid [labs/private] - 10https://gerrit.wikimedia.org/r/1059029 (https://phabricator.wikimedia.org/T360636)
[10:13:56] <wikibugs>	 (03update) 10sstefanova: api: add boilerplate for endpoints [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/4 (https://phabricator.wikimedia.org/T362069)
[10:21:23] <wikibugs>	 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops: MVP: Privately server wiktech via mw-on-k8s - https://phabricator.wikimedia.org/T371537#10034765 (10jijiki) >>! In T371537#10033727, @bd808 wrote: > The config thing that most needs to be changed to use the multiversion images is `/etc/mediawiki/WikitechPr...
[10:32:17] <wikibugs>	 (03update) 10sstefanova: api: add boilerplate for endpoints [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/4 (https://phabricator.wikimedia.org/T362069)
[10:38:11] <wikibugs>	 10Toolforge: Please re-install "joe" - https://phabricator.wikimedia.org/T371556#10034811 (10Aklapper) a:05Plenz→03None
[10:52:34] <wikibugs>	 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations: Allow wikitech to use its own internal authentication - wikitech self-auth - https://phabricator.wikimedia.org/T371588 (10jijiki) 03NEW
[10:54:15] <wikibugs>	 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations: Allow wikitech to use its own internal authentication - wikitech self-auth - https://phabricator.wikimedia.org/T371588#10034840 (10jijiki)
[10:54:49] <wikibugs>	 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops: ☂ Migrate Wikitech to Kubernetes - https://phabricator.wikimedia.org/T292707#10034841 (10jijiki)
[10:55:28] <wikibugs>	 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops: ☂ Migrate Wikitech to Kubernetes - https://phabricator.wikimedia.org/T292707#10034843 (10jijiki)
[10:55:32] <wikibugs>	 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10034842 (10jijiki)
[10:59:51] <wikibugs>	 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops: MVP: Privately serve wikitech via mw-on-k8s - https://phabricator.wikimedia.org/T371537#10034846 (10akosiaris)
[11:02:54] <wikibugs>	 10Toolforge: Please re-install "joe" - https://phabricator.wikimedia.org/T371556#10034860 (10LucasWerkmeister) 05Resolved→03Open No, staying on an already-discontinued operating system version is not a permanent solution.
[11:13:37] <wikibugs>	 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations: wikitech self-auth: Allow wikitech to use its own internal authentication - https://phabricator.wikimedia.org/T371588#10034880 (10jijiki) p:05Triage→03High
[11:14:34] <wikibugs>	 (03open) 10dcaro: Draft: show backend status [toolforge-repos/sample-complex-app-frontend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-frontend/-/merge_requests/1 (https://phabricator.wikimedia.org/T370324)
[11:16:51] <wikibugs>	 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations: LdapAuthentication: Disable extension from Wikitech - https://phabricator.wikimedia.org/T371592 (10jijiki) 03NEW
[11:23:22] <wikibugs>	 (03open) 10sstefanova: get_versions: add components-api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/465
[11:23:23] <wikibugs>	 10Toolforge: Please re-install "joe" - https://phabricator.wikimedia.org/T371556#10034913 (10Plenz) OK, so please install "joe" on the permant operating system and then tell me how to login.
[11:26:03] <wikibugs>	 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations: Developer Ban: Migrate the one-stop Developer ban functionality away from Wikitech - https://phabricator.wikimedia.org/T371593 (10jijiki) 03NEW
[11:29:29] <wikibugs>	 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 06serviceops: Developer Account Blocking: Migrate the one-stop Developer ban functionality away from Wikitech - https://phabricator.wikimedia.org/T371593#10034931 (10jijiki)
[11:29:39] <wikibugs>	 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 06serviceops: LdapAuthentication: Disable extension from Wikitech - https://phabricator.wikimedia.org/T371592#10034936 (10jijiki)
[11:30:09] <wikibugs>	 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 06serviceops: Developer Account Blocking: Migrate the one-stop Developer ban functionality away from Wikitech - https://phabricator.wikimedia.org/T371593#10034934 (10jijiki)
[11:30:41] <wikibugs>	 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 06serviceops: wikitech self-auth: Allow wikitech to use its own internal authentication - https://phabricator.wikimedia.org/T371588#10034938 (10jijiki)
[11:43:15] <wikibugs>	 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 06serviceops: Developer Account Blocking: Migrate the one-stop Developer ban functionality away from Wikitech - https://phabricator.wikimedia.org/T371593#10034963 (10Bugreporter)
[11:45:30] <wikibugs>	 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops: Migrate Wikitech's Jobqueue - https://phabricator.wikimedia.org/T371359#10034971 (10jijiki)
[11:49:54] <wikibugs>	 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 06serviceops: Developer Account Blocking: Migrate the one-stop Developer (un)Block functionality away from Wikitech - https://phabricator.wikimedia.org/T371593#10034980 (10jijiki)
[11:59:16] <wikibugs>	 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 06serviceops: Developer Account Blocking: Migrate the one-stop Developer (un)Block functionality away from Wikitech - https://phabricator.wikimedia.org/T371593#10034990 (10jijiki) →14Duplicate dup:03T359820
[11:59:17] <wikibugs>	 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 06serviceops: Developer Account Blocking: Migrate the one-stop Developer (un)Block functionality away from Wikitech - https://phabricator.wikimedia.org/T371593#10034995 (10jijiki)
[11:59:21] <wikibugs>	 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 06serviceops: LdapAuthentication: Disable extension from Wikitech - https://phabricator.wikimedia.org/T371592#10034996 (10jijiki)
[11:59:32] <wikibugs>	 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10034997 (10jijiki)
[12:02:46] <wikibugs>	 10Striker: Remove feature to connect SUL account to Striker - https://phabricator.wikimedia.org/T371595 (10Bugreporter) 03NEW
[12:21:01] <wikibugs>	 (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/19
[12:43:11] <wikibugs>	 (03PS1) 10Andrew Bogott: Fake rabbitmq passwords for designate [labs/private] - 10https://gerrit.wikimedia.org/r/1059065 (https://phabricator.wikimedia.org/T320256)
[12:44:11] <wikibugs>	 (03PS2) 10Andrew Bogott: Fake rabbitmq passwords for designate [labs/private] - 10https://gerrit.wikimedia.org/r/1059065 (https://phabricator.wikimedia.org/T320256)
[12:51:04] <wikibugs>	 (03CR) 10Andrew Bogott: [V:03+2 C:03+2] Fake rabbitmq passwords for designate [labs/private] - 10https://gerrit.wikimedia.org/r/1059065 (https://phabricator.wikimedia.org/T320256) (owner: 10Andrew Bogott)
[12:54:01] <wikibugs>	 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 06serviceops: Developer Account Blocking: Migrate the one-stop Developer (un)Block functionality away from Wikitech - https://phabricator.wikimedia.org/T371593#10035257 (10jijiki)
[12:54:07] <wikibugs>	 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 06serviceops: LdapAuthentication: Disable extension from Wikitech - https://phabricator.wikimedia.org/T371592#10035258 (10jijiki)
[12:59:16] <wikibugs>	 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops: Migrate Wikitech's Jobqueue - https://phabricator.wikimedia.org/T371359#10035291 (10jijiki)
[13:09:18] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack
[13:09:41] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
[13:17:34] <wikibugs>	 10PAWS: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T371337#10035347 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/443
[13:17:38] <wikibugs>	 10Tools, 10Wikidata, 10wmde-wikidata-tech, 10Wikidata Dev Team (Wikidata.org Slice): [GENERAL] Deprecate connecting senses prototype - https://phabricator.wikimedia.org/T351829#10035340 (10Lucas_Werkmeister_WMDE) 05Open→03Resolved a:03ArthurTaylor
[13:17:42] <wikibugs>	 10PAWS: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T371337#10035348 (10rook) 05Open→03Resolved a:03rook
[13:17:49] <notefromgithub>	 vivian-rook closed https://github.com/toolforge/paws/pull/443
[13:36:12] <wikibugs>	 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops: Memcached: Migrate wikitech to main memcached - https://phabricator.wikimedia.org/T371608 (10jijiki) 03NEW
[13:38:47] <wikibugs>	 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 06serviceops: wikitech self-auth: Allow wikitech to use its own internal authentication - https://phabricator.wikimedia.org/T371588#10035449 (10jijiki) a:03Ladsgroup
[13:44:01] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack
[13:44:20] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
[13:47:24] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Give openstack services their own rabbitmq users - https://phabricator.wikimedia.org/T320256#10035487 (10Andrew) Now we have per-service users for each openstack service.
[13:51:35] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Give openstack services their own rabbitmq users - https://phabricator.wikimedia.org/T320256#10035495 (10Andrew) 05Open→03Resolved
[14:09:41] <wikibugs>	 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops: MVP: Privately serve wikitech via mw-on-k8s - https://phabricator.wikimedia.org/T371537#10035577 (10jijiki)
[14:09:42] <wikibugs>	 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops: Apache: Include Wikitech in mw-on-k8s' virtual hosts - https://phabricator.wikimedia.org/T371360#10035578 (10jijiki)
[14:14:29] <wikibugs>	 (03approved) 10sstefanova: get_versions: add components-api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/465
[14:14:32] <wikibugs>	 (03merge) 10sstefanova: get_versions: add components-api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/465
[14:18:22] <wikibugs>	 (03approved) 10sstefanova: api: add boilerplate for endpoints [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/4 (https://phabricator.wikimedia.org/T362069)
[14:18:27] <wikibugs>	 (03merge) 10sstefanova: api: add boilerplate for endpoints [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/4 (https://phabricator.wikimedia.org/T362069)
[14:20:48] <wikibugs>	 (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: components-api: bump to 0.0.5-20240801141838-47c73802 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/466 (https://phabricator.wikimedia.org/T362069)
[14:21:09] <wikibugs>	 10Striker, 10Bitu, 06Infrastructure-Foundations: Remove feature to connect SUL account to Striker (read from Bitu instead) - https://phabricator.wikimedia.org/T371595#10035602 (10Bugreporter)
[14:23:03] <wikibugs>	 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Migrate deployment-prep away from Debian Buster to Bullseye/Bookworm - https://phabricator.wikimedia.org/T327742#10035620 (10Andrew)
[14:23:55] <wikibugs>	 10cloud-services-team (FY2023/2024-Q3-Q4), 10superset.wmcloud.org: Allow Superset to query ToolsDB public databases - https://phabricator.wikimedia.org/T367393#10035621 (10github-toolforge-bot) dhinus opened https://github.com/toolforge/superset-deploy/pull/27
[14:24:07] <notefromgithub>	 dhinus opened https://github.com/toolforge/superset-deploy/pull/27
[14:27:24] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Remove or replace poolcounter06.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370458#10035634 (10Andrew)
[14:27:47] <wikibugs>	 (03update) 10sstefanova: functional-tests: add direct-api tests for components api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/464
[14:27:52] <wikibugs>	 (03approved) 10sstefanova: functional-tests: add direct-api tests for components api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/464
[14:27:55] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Remove or replace poolcounter06.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370458#10035635 (10Andrew) Oh, I see that the prod poolcounter hosts are also still r...
[14:28:20] <wikibugs>	 (03update) 10sstefanova: functional-tests: add direct-api tests for components api [repos/cloud/toolforge/toolforge-deploy] (bump_components-api) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/464
[14:30:41] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Remove or replace deployment-restbase04.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370460#10035657 (10Andrew)
[14:30:48] <wikibugs>	 (03update) 10sstefanova: functional-tests: add direct-api tests for components api [repos/cloud/toolforge/toolforge-deploy] (bump_components-api) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/464
[14:31:47] <wikibugs>	 (03update) 10sstefanova: functional-tests: add direct-api tests for components api [repos/cloud/toolforge/toolforge-deploy] (bump_components-api) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/464
[14:32:14] <wikibugs>	 (03merge) 10sstefanova: functional-tests: add direct-api tests for components api [repos/cloud/toolforge/toolforge-deploy] (bump_components-api) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/464
[14:32:17] <wikibugs>	 (03update) 10sstefanova: components-api: bump to 0.0.5-20240801141838-47c73802 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/466 (https://phabricator.wikimedia.org/T362069) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[14:47:09] <notefromgithub>	 dhinus opened https://github.com/toolforge/quarry/pull/61
[14:47:11] <wikibugs>	 10cloud-services-team (FY2023/2024-Q3-Q4), 10Quarry: Allow Quarry to query its own database - https://phabricator.wikimedia.org/T367415#10035700 (10github-toolforge-bot) dhinus opened https://github.com/toolforge/quarry/pull/61
[14:52:49] <wikibugs>	 10cloud-services-team (FY2023/2024-Q3-Q4), 10superset.wmcloud.org: Allow Superset to query ToolsDB public databases - https://phabricator.wikimedia.org/T367393#10035716 (10Zache) I got an error when i tried to query bigint columns  > MySQL Error: Python int too large to convert to C long  How to reproduce:   #...
[14:55:46] <wikibugs>	 10cloud-services-team (FY2023/2024-Q3-Q4), 10superset.wmcloud.org: Allow Superset to query ToolsDB public databases - https://phabricator.wikimedia.org/T367393#10035718 (10fnegri) @Zache the query works for me:  {F56878752}
[15:08:21] <wikibugs>	 10cloud-services-team (FY2023/2024-Q3-Q4), 10superset.wmcloud.org: Allow Superset to query ToolsDB public databases - https://phabricator.wikimedia.org/T367393#10035747 (10Zache) I updated the SQL query so that it will fetch large numbers instead of zeroes. (ie. SELECT max(phash) FROM imagehash) Also screensho...
[15:23:48] <wikibugs>	 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 06serviceops: LdapAuthentication: Disable extension from Wikitech - https://phabricator.wikimedia.org/T371592#10035804 (10jijiki) a:03jijiki
[15:26:32] <logmsgbot_cloud>	 !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api
[15:26:42] <logmsgbot_cloud>	 !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
[15:27:22] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Rebuild or delete  deployment-docker-cpjobqueue01 - https://phabricator.wikimedia.org/T369914#10035808 (10Urbanecm_WMF) >>! In T369914#9997770, @Southparkfan wrote: > Instance is offline, seems to be superseded by...
[15:30:46] <wikibugs>	 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Migrate deployment-prep away from Debian Buster to Bullseye/Bookworm - https://phabricator.wikimedia.org/T327742#10035833 (10Andrew)
[15:31:13] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Rebuild or delete  deployment-docker-cpjobqueue01 - https://phabricator.wikimedia.org/T369914#10035828 (10Andrew) 05Open→03Resolved a:03Andrew deleted!
[15:31:55] <logmsgbot_cloud>	 !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api
[15:32:05] <logmsgbot_cloud>	 !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
[15:32:37] <wikibugs>	 10cloud-services-team (FY2023/2024-Q3-Q4), 10superset.wmcloud.org: Allow Superset to query ToolsDB public databases - https://phabricator.wikimedia.org/T367393#10035846 (10fnegri) @Zache hmm this looks like a Superset bug to me, I found https://github.com/apache/superset/issues/15876 but they say it's resolved...
[15:38:25] <wikibugs>	 (03update) 10sstefanova: components-api: bump to 0.0.5-20240801141838-47c73802 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/466 (https://phabricator.wikimedia.org/T362069) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[15:38:26] <wikibugs>	 (03approved) 10sstefanova: components-api: bump to 0.0.5-20240801141838-47c73802 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/466 (https://phabricator.wikimedia.org/T362069) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[15:39:08] <wikibugs>	 (03merge) 10sstefanova: components-api: bump to 0.0.5-20240801141838-47c73802 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/466 (https://phabricator.wikimedia.org/T362069) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[15:43:24] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 13), 13Patch-For-Review: toolforge: upgrade all Kubernetes components to versions supporting Kubernetes 1.26 - https://phabricator.wikimedia.org/T370046#10035894 (10Slst2020) 05In progress→03Stalled
[15:43:26] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 13), 13Patch-For-Review: toolforge: upgrade all Kubernetes components to versions supporting Kubernetes 1.26 - https://phabricator.wikimedia.org/T370046#10035897 (10Slst2020) stalled on calico upgrade https://gitlab.wikimedia.org/repos/cloud/toolforge/t...
[15:56:11] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Remove or replace poolcounter06.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370458#10035969 (10Dzahn) That would be T332015 and a question for the serviceops team.
[15:56:36] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Remove or replace poolcounter06.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370458#10035971 (10Dzahn)
[16:00:35] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Remove or replace poolcounter06.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370458#10035995 (10Dzahn) @Andrew Good news, see T332015#10035989
[16:02:41] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Remove or replace poolcounter06.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370458#10036010 (10Andrew) Oh, great. @Southparkfan does that unblock you here?
[16:19:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[18:38:25] <wikibugs>	 (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/19 (owner: 10l10n-bot)
[18:38:28] <wikibugs>	 (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/19 (owner: 10l10n-bot)
[18:52:56] <wikibugs>	 10Cloud-VPS, 06collaboration-services, 13Patch-For-Review: puppet problems mounting cinder volumes (and suggested fixes) - https://phabricator.wikimedia.org/T371573#10036550 (10Dzahn)
[18:53:25] <wikibugs>	 10Cloud-VPS (Quota-requests), 06collaboration-services: Increase quota for volumes in devtools project - https://phabricator.wikimedia.org/T371093#10036548 (10Dzahn) Thank you @Slst2020
[19:07:08] <mutante>	 !log devtools creating new 20GB volume 'gitlab-prod-logs' T371066
[19:07:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL
[19:07:12] <stashbot>	 T371066: gitlab/devtools: send logs to a new disk - https://phabricator.wikimedia.org/T371066
[19:11:32] <mutante>	 !log devtools attaching volume 'gitlab-prod-logs' to instance 'gitlab-prod-1002' T371066 | running 'sudo wmcs-prepare-cinder-volume' manually and answering interactive questions to mount/format it  (once T371573 is resolved will do this with puppet)
[19:11:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL
[19:11:36] <stashbot>	 T371573: puppet problems mounting cinder volumes (and suggested fixes) - https://phabricator.wikimedia.org/T371573
[19:18:41] <mutante>	 !log devtools gitlab-prod-1002 - moved contents of /var/log/gitlab out of the way temp, emptied it, letting wmcs-prepare-cinder-volume mount new volume into /var/log/gitlab and format it with ext4, moved old existing logs back into it  T371066
[19:18:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL
[19:18:49] <stashbot>	 T371066: gitlab/devtools: send logs to a new disk - https://phabricator.wikimedia.org/T371066
[19:24:15] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Upgrade cloud-vps openstack to version 'Caracal' - https://phabricator.wikimedia.org/T369044#10036624 (10Andrew) a:03Andrew
[19:28:48] <wikibugs>	 (03open) 10raymond-ndibe: [jobs-api] better error handling for services [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/113 (https://phabricator.wikimedia.org/T359804)
[19:28:56] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] better error handling for services [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/113 (https://phabricator.wikimedia.org/T359804)
[19:29:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 5 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[19:51:03] <wmcs-alerts>	 FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-3 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses
[19:56:03] <wmcs-alerts>	 RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-3 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses
[20:32:10] <logmsgbot_cloud>	 !log bd808@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-55
[20:37:32] <logmsgbot_cloud>	 !log bd808@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-55
[22:41:28] <wmcs-alerts>	 FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance gitlab-runners-puppetserver-01 in project gitlab-runners   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure
[22:53:30] <wikibugs>	 10Tools: Transfer ownership of SpinachBot from HTriedman (WMF) to HTriedman - https://phabricator.wikimedia.org/T371644#10037122 (10Aklapper)
[23:00:18] <wikibugs>	 10Tools: Transfer ownership of SpinachBot from HTriedman (WMF) to HTriedman - https://phabricator.wikimedia.org/T371644#10037132 (10Dzahn) Are you planning to stay under volunteer NDA to keep access to private tickets?  (membership in WMF-NDA group in Phab)  Did you want to keep various web-based logins (tied to...