[00:02:07] <wmcs-alerts>	 RESOLVED: ToolsDBHistoryLengthGrowing: ToolsDB History Length is above the desired threshold - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBHistoryLengthGrowing  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBHistoryLengthGrowing
[00:31:24] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: hdfs_rsync_mediawiki_content_history.service on clouddumps1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[00:32:46] <wikibugs>	 10Cloud-VPS, 06tools-infrastructure-team: Resize legacy.wikiapiary.eqiad1.wikimedia.cloud to g4.core16.ram32.disk20 - https://phabricator.wikimedia.org/T439317#12398564 (10MarkAHershberger) This disk can be removed.  If it was removed could we then resize the VM normally?
[00:36:25] <jinxer-wm>	 FIRING: [3x] SystemdUnitFailed: backup_cinder_volumes.service on cloudbackup2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[00:58:44] <jinxer-wm>	 FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors
[01:33:44] <jinxer-wm>	 RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors
[03:00:57] <wikibugs>	 (03open) 10raymond-ndibe: models: split filelog options from common run fields [repos/cloud/toolforge/components-api] (fix_component_type_discriminator) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/198 (https://phabricator.wikimedia.org/T362077)
[03:01:34] <wikibugs>	 (03open) 10raymond-ndibe: feat: add webservice component type support [repos/cloud/toolforge/components-api] (split_filelog_options_from_run_info_fields) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/199 (https://phabricator.wikimedia.org/T362077)
[03:40:57] <jinxer-wm>	 FIRING: ProbeDown: Service labweb-ssl:7443 has failed probes (http_labweb-ssl_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#labweb-ssl:7443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[03:45:57] <jinxer-wm>	 RESOLVED: ProbeDown: Service labweb-ssl:7443 has failed probes (http_labweb-ssl_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#labweb-ssl:7443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[04:36:25] <jinxer-wm>	 FIRING: [3x] SystemdUnitFailed: backup_cinder_volumes.service on cloudbackup2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[07:05:10] <wikibugs>	 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Ensure cloudvirt capacity is more evenly spread out among racks - https://phabricator.wikimedia.org/T424658#12399002 (10fgiunchedi)
[07:06:25] <jinxer-wm>	 FIRING: [3x] SystemdUnitFailed: backup_cinder_volumes.service on cloudbackup2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[08:12:58] <wikibugs>	 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [components-api] pick component model based on component-type field, not data shape - https://phabricator.wikimedia.org/T439496#12399361 (10dcaro) > so I played with this even more and while the publish approach is nice, it does prevent...
[08:20:42] <wikibugs>	 (03update) 10dcaro: global: adopt shared standards [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/72 (https://phabricator.wikimedia.org/T440184)
[08:21:10] <wikibugs>	 (03update) 10dcaro: global: adopt shared standards [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/72 (https://phabricator.wikimedia.org/T440184)
[09:19:05] <wikibugs>	 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Ensure cloudvirt capacity is more evenly spread out among racks - https://phabricator.wikimedia.org/T424658#12399593 (10fgiunchedi) @Jclark-ctr @VRiley-WMF we'll be moving four cloudvirt out of D5 and into C...
[09:21:38] <wikibugs>	 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops: Set cloudvirt fleet to 'performance' governor - https://phabricator.wikimedia.org/T439729#12399602 (10fgiunchedi)
[09:36:24] <wikibugs>	 (03PS1) 10Btullis: idp_test: Add a dummy client secret for wmf_s3_login [labs/private] - 10https://gerrit.wikimedia.org/r/1351958 (https://phabricator.wikimedia.org/T435602)
[09:37:21] <wikibugs>	 (03CR) 10Btullis: [V:03+2 C:03+2] idp_test: Add a dummy client secret for wmf_s3_login [labs/private] - 10https://gerrit.wikimedia.org/r/1351958 (https://phabricator.wikimedia.org/T435602) (owner: 10Btullis)
[10:57:10] <wikibugs>	 10Tool-inteGraality: Retire property_threshold parameter - https://phabricator.wikimedia.org/T440290 (10JeanFred) 03NEW
[11:06:17] <wmcs-alerts>	 FIRING: PrometheusRestarted: Prometheus instance tools-prometheus-8:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted
[11:06:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: backup_cinder_volumes.service on cloudbackup2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[11:36:17] <wmcs-alerts>	 RESOLVED: PrometheusRestarted: Prometheus instance tools-prometheus-8:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted
[11:47:17] <jinxer-wm>	 FIRING: JobUnavailable: Reduced availability for job blackbox_http in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[11:52:17] <jinxer-wm>	 RESOLVED: JobUnavailable: Reduced availability for job blackbox_http in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[11:58:25] <wikibugs>	 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12400125 (10fgiunchedi) With T439831 in place I was able to reimage 1055 and 1056 for their initial puppet runs. All disks are exposed as JBOD for these an...
[12:02:09] <wikibugs>	 10Toolforge, 06tools-platform-team: [maintain-harbor] add pagination when querying harbor - https://phabricator.wikimedia.org/T440294 (10dcaro) 03NEW
[12:03:35] <wikibugs>	 10Toolforge, 06tools-platform-team: [maintain-harbor] add 'noop' mode for testing - https://phabricator.wikimedia.org/T440295 (10dcaro) 03NEW
[12:03:45] <wikibugs>	 10Toolforge, 06tools-platform-team: [maintain-harbor] add pagination when querying harbor - https://phabricator.wikimedia.org/T440294#12400163 (10dcaro) p:05Triage→03High
[12:29:11] <wikibugs>	 10Tool-delintbot: Store data about blocked edit attempts - https://phabricator.wikimedia.org/T440299 (10Redmin) 03NEW
[12:29:21] <wikibugs>	 10Tool-delintbot: Store data about blocked edit attempts - https://phabricator.wikimedia.org/T440299#12400286 (10Redmin) p:05Triage→03Medium
[12:29:53] <wikibugs>	 10Toolforge, 06tools-platform-team: [maintain-harbor] add 'noop' mode for testing - https://phabricator.wikimedia.org/T440295#12400289 (10dcaro) p:05Triage→03High
[12:29:59] <wikibugs>	 10Toolforge, 06tools-platform-team: [maintain-harbor] add 'noop' mode for testing - https://phabricator.wikimedia.org/T440295#12400291 (10dcaro) 05Open→03In progress
[12:30:03] <wikibugs>	 10Toolforge, 06tools-platform-team: [maintain-harbor] add pagination when querying harbor - https://phabricator.wikimedia.org/T440294#12400294 (10dcaro) 05Open→03In progress
[12:35:26] <wikibugs>	 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12400316 (10fgiunchedi) With respect to 1053 and 1054 I would like to get those in their final position, which brings up to the rack power question.  In {T...
[12:50:38] <wikibugs>	 10Data-Services, 06tools-infrastructure-team, 06tools-platform-team: clouddb memory alert - https://phabricator.wikimedia.org/T438200#12400373 (10fnegri) 05In progress→03Resolved This alert has not triggered since 2026-09-21, so I'm optimistically marking this as Resolved.  Something I noticed is tha...
[13:22:43] <wikibugs>	 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387#12400562 (10Andrew) >>! In T429387#12394141, @Andrew wrote: > Moving 1043 to from 32k to 16k allocation since it doesn't s...
[13:30:36] <wikibugs>	 (03update) 10raymond-ndibe: feat: add webservice component type support [repos/cloud/toolforge/components-api] (split_filelog_options_from_run_info_fields) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/199 (https://phabricator.wikimedia.org/T362077)
[13:31:25] <jinxer-wm>	 RESOLVED: SystemdUnitFailed: backup_cinder_volumes.service on cloudbackup2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[13:40:05] <wikibugs>	 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [components-api] pick component model based on component-type field, not data shape - https://phabricator.wikimedia.org/T439496#12400755 (10Raymond_Ndibe) >>! In T439496#12399361, @dcaro wrote: >> so I played with this even more and whi...
[13:42:31] <wikibugs>	 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [components-api] pick component model based on component-type field, not data shape - https://phabricator.wikimedia.org/T439496#12400765 (10Raymond_Ndibe) > I'm going with using a different field name like webservice_command or argument...
[13:42:44] <wikibugs>	 (03open) 10fnegri: Read from components-api which tools opted in [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/12 (https://phabricator.wikimedia.org/T439554)
[13:44:20] <wikibugs>	 (03close) 10raymond-ndibe: config: change config schema to v1beta2 while maintaining v1beta1 support [repos/cloud/toolforge/components-api] (fix_component_type_discriminator) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/194 (https://phabricator.wikimedia.org/T439496)
[14:03:05] <wikibugs>	 (03update) 10raymond-ndibe: models: make component_type the component union discriminator [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/195 (https://phabricator.wikimedia.org/T439496)
[14:03:20] <wikibugs>	 (03update) 10raymond-ndibe: models: split filelog options from common run fields [repos/cloud/toolforge/components-api] (fix_component_type_discriminator) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/198 (https://phabricator.wikimedia.org/T362077)
[14:03:26] <wikibugs>	 (03update) 10raymond-ndibe: feat: add webservice component type support [repos/cloud/toolforge/components-api] (split_filelog_options_from_run_info_fields) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/199 (https://phabricator.wikimedia.org/T362077)
[14:03:56] <wikibugs>	 (03update) 10fnegri: Read from components-api which tools opted in [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/12 (https://phabricator.wikimedia.org/T439554)
[14:03:58] <wikibugs>	 (03update) 10fnegri: Read from components-api which tools opted in [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/12 (https://phabricator.wikimedia.org/T439554)
[14:05:21] <wikibugs>	 (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/196 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[14:06:28] <wikibugs>	 (03update) 10raymond-ndibe: models: make component_type the component union discriminator [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/195 (https://phabricator.wikimedia.org/T439496)
[14:06:38] <wikibugs>	 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [components-api] pick component model based on component-type field, not data shape - https://phabricator.wikimedia.org/T439496#12401039 (10dcaro) >>! In T439496#12400755, @Raymond_Ndibe wrote: >>>! In T439496#12399361, @dcaro wrote: >>...
[14:08:29] <wikibugs>	 (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.221-20261006140537-10ef2890 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1451
[14:08:34] <wikibugs>	 (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.221-20261006140537-10ef2890 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1451
[14:19:12] <wikibugs>	 10Toolforge, 06tools-platform-team, 07OKR-Work: [components-api] Allow configs without any component - https://phabricator.wikimedia.org/T440326 (10fnegri) 03NEW
[14:24:47] <wikibugs>	 (03approved) 10raymond-ndibe: global: adopt shared standards [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/72 (https://phabricator.wikimedia.org/T440184) (owner: 10dcaro)
[14:28:15] <wikibugs>	 (03open) 10fnegri: basic_system: Keep the inotify limits after a reboot [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/344 (https://phabricator.wikimedia.org/T429347)
[14:31:59] <wikibugs>	 (03approved) 10raymond-ndibe: basic_system: Keep the inotify limits after a reboot [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/344 (https://phabricator.wikimedia.org/T429347) (owner: 10fnegri)
[14:34:17] <wikibugs>	 10Toolforge, 06tools-platform-team: [lima-kilo] inotify settings are lost after a VM restart - https://phabricator.wikimedia.org/T440333 (10fnegri) 03NEW
[14:34:25] <wikibugs>	 (03update) 10fnegri: basic_system: Keep the inotify limits after a reboot [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/344 (https://phabricator.wikimedia.org/T429347 https://phabricator.wikimedia.org/T440333)
[14:34:31] <wikibugs>	 (03update) 10fnegri: basic_system: Keep the inotify limits after a reboot [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/344 (https://phabricator.wikimedia.org/T440333)
[14:38:30] <wikibugs>	 (03merge) 10fnegri: basic_system: Keep the inotify limits after a reboot [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/344 (https://phabricator.wikimedia.org/T440333)
[14:39:58] <wikibugs>	 10Toolforge, 06tools-platform-team: [lima-kilo] inotify settings are lost after a VM restart - https://phabricator.wikimedia.org/T440333#12401333 (10fnegri) 05Open→03Resolved a:03fnegri
[14:46:53] <wikibugs>	 (03open) 10fnegri: config: allow configs without components [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/200 (https://phabricator.wikimedia.org/T440326)
[14:50:30] <wikibugs>	 10Toolforge, 06tools-platform-team: [lima-kilo] increase the number of inotify watches - https://phabricator.wikimedia.org/T429347#12401470 (10fnegri)
[14:51:44] <jinxer-wm>	 FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors
[15:06:43] <wikibugs>	 10Toolforge, 06tools-platform-team: [maintain-harbor] delete untagged artifacts too - https://phabricator.wikimedia.org/T440343 (10dcaro) 03NEW
[15:06:50] <wikibugs>	 10Toolforge, 06tools-platform-team: [toolsbeta,harbor] Got out of quota for CI images (toolforge project) - https://phabricator.wikimedia.org/T440182#12401596 (10dcaro)
[15:06:51] <wikibugs>	 10Toolforge, 06tools-platform-team: [maintain-harbor] delete untagged artifacts too - https://phabricator.wikimedia.org/T440343#12401597 (10dcaro)
[15:28:13] <wikibugs>	 (03merge) 10dcaro: global: adopt shared standards [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/72 (https://phabricator.wikimedia.org/T440184)
[15:28:21] <wikibugs>	 (03open) 10dcaro: add noop [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/73 (https://phabricator.wikimedia.org/T440295)
[15:28:35] <wikibugs>	 (03open) 10dcaro: jobs: delete also untagged artifacts [repos/cloud/toolforge/maintain-harbor] (add_noop) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/74 (https://phabricator.wikimedia.org/T440343)
[15:30:17] <wikibugs>	 (03open) 10dcaro: add pagination [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/75 (https://phabricator.wikimedia.org/T440294)
[15:30:33] <wikibugs>	 (03update) 10dcaro: add noop [repos/cloud/toolforge/maintain-harbor] (add_pagination) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/73 (https://phabricator.wikimedia.org/T440295)
[15:30:37] <wikibugs>	 (03update) 10dcaro: add noop [repos/cloud/toolforge/maintain-harbor] (add_pagination) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/73 (https://phabricator.wikimedia.org/T440295)
[15:36:44] <jinxer-wm>	 RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors
[15:37:40] <wikibugs>	 (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: maintain-harbor: bump to 0.0.77-20261006153234-e690a788 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1447 (https://phabricator.wikimedia.org/T440184)
[15:37:50] <wikibugs>	 (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: maintain-harbor: bump to 0.0.77-20261006153234-e690a788 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1447 (https://phabricator.wikimedia.org/T440184)
[15:38:27] <wikibugs>	 (03update) 10dcaro: add pagination [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/75 (https://phabricator.wikimedia.org/T440294)
[15:40:41] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
[15:42:32] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
[15:43:20] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
[15:47:11] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor
[15:48:44] <wikibugs>	 (03approved) 10dcaro: maintain-harbor: bump to 0.0.77-20261006153234-e690a788 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1447 (https://phabricator.wikimedia.org/T440184) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[15:48:49] <wikibugs>	 (03merge) 10dcaro: maintain-harbor: bump to 0.0.77-20261006153234-e690a788 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1447 (https://phabricator.wikimedia.org/T440184) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[15:49:33] <wikibugs>	 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [maintain-harbor] delete untagged artifacts too - https://phabricator.wikimedia.org/T440343#12401954 (10dcaro) p:05Triage→03High
[15:49:38] <wikibugs>	 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [components-api] Allow configs without any component - https://phabricator.wikimedia.org/T440326#12401955 (10dcaro) p:05Triage→03High
[15:50:00] <wikibugs>	 10Toolforge, 06tools-platform-team: [maintain-harbor] package with poetry/etc. like the rest - https://phabricator.wikimedia.org/T440184#12401957 (10dcaro) 05In progress→03Resolved
[15:52:18] <wikibugs>	 10Cloud-VPS, 06tools-infrastructure-team: magnum-cluster-api: understand and back up tenant cluster state - https://phabricator.wikimedia.org/T440363 (10Andrew) 03NEW
[16:26:47] <wikibugs>	 (03open) 10dcaro: maintain-harbor: adapt message to new output [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1457 (https://phabricator.wikimedia.org/T440343)
[16:28:24] <wikibugs>	 (03update) 10fnegri: Read from components-api which tools opted in [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/12 (https://phabricator.wikimedia.org/T439554)
[16:28:26] <wikibugs>	 (03update) 10fnegri: Read from components-api which tools opted in [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/12 (https://phabricator.wikimedia.org/T439554)
[16:29:36] <wikibugs>	 (03approved) 10dcaro: config: allow configs without components [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/200 (https://phabricator.wikimedia.org/T440326) (owner: 10fnegri)
[16:29:54] <wikibugs>	 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [components-api] Allow configs without any component - https://phabricator.wikimedia.org/T440326#12402448 (10dcaro) 05Open→03In progress
[16:31:11] <wikibugs>	 (03update) 10dcaro: models: make component_type the component union discriminator [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/195 (https://phabricator.wikimedia.org/T439496) (owner: 10raymond-ndibe)
[16:33:43] <wikibugs>	 (03update) 10fnegri: Read from components-api which tools opted in [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/12 (https://phabricator.wikimedia.org/T439554)
[16:36:14] <wikibugs>	 (03update) 10dcaro: models: split filelog options from common run fields [repos/cloud/toolforge/components-api] (fix_component_type_discriminator) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/198 (https://phabricator.wikimedia.org/T362077) (owner: 10raymond-ndibe)
[16:42:49] <wikibugs>	 (03approved) 10dcaro: Read from components-api which tools opted in [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/12 (https://phabricator.wikimedia.org/T439554) (owner: 10fnegri)
[16:43:30] <wikibugs>	 10Tool-plantilla-chatbot, 06Release-Engineering-Team, 10GitLab (Project and group requests): GitLab Private Repository Request for: pcendrer/plantilla-chatbot-patrones - https://phabricator.wikimedia.org/T440370#12402586 (10brennen) a:03brennen Could you describe the plantilla-chatbot project?
[16:43:31] <wikibugs>	 (03update) 10dcaro: maintain-kubeusers: bump to 0.0.203-20261005123458-89615cdc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1452 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[16:43:42] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
[16:44:54] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers
[16:46:59] <wikibugs>	 10Toolforge, 06tools-platform-team: [functional-tests,deploy] Lately got ~50% errors due to not finding the branch - https://phabricator.wikimedia.org/T440224#12402606 (10dcaro) Happened again, it did not help much xd  ` @@@@@@@@ Configuring toolforge-deploy for toolsbeta.test + git branch -a + grep -qwE 'bump...
[16:48:20] <wikibugs>	 (03update) 10dcaro: maintain-kubeusers: bump to 0.0.203-20261005123458-89615cdc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1452 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[16:48:27] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
[16:55:30] <wikibugs>	 10Tool-plantilla-chatbot, 06Release-Engineering-Team, 10GitLab (Project and group requests): GitLab Private Repository Request for: pcendrer/plantilla-chatbot-patrones - https://phabricator.wikimedia.org/T440370#12402664 (10PCendrer) plantilla-chatbot is a spanish wikipedia bot for fight against the AI in th...
[16:56:29] <wikibugs>	 10Tool-plantilla-chatbot, 06Release-Engineering-Team, 10GitLab (Project and group requests): GitLab Private Repository Request for: pcendrer/plantilla-chatbot-patrones - https://phabricator.wikimedia.org/T440370#12402671 (10Aklapper)
[16:57:54] <wikibugs>	 (03update) 10dcaro: ci: use trixie [toolforge-repos/k8s-status] - 10https://gitlab.wikimedia.org/toolforge-repos/k8s-status/-/merge_requests/8
[16:59:52] <wikibugs>	 10Tool-plantilla-chatbot, 06Release-Engineering-Team, 07Essential-Work, 10GitLab (Project and group requests): GitLab Private Repository Request for: pcendrer/plantilla-chatbot-patrones - https://phabricator.wikimedia.org/T440370#12402693 (10brennen) 05Open→03Resolved Thanks - it sounds like this f...
[17:04:07] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
[17:05:09] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
[17:06:11] <wikibugs>	 (03update) 10raymond-ndibe: models.py: expose state and webservice_defaults in images endpoint [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/347 (https://phabricator.wikimedia.org/T431679)
[17:07:53] <wikibugs>	 (03update) 10raymond-ndibe: models.py: expose state and webservice_defaults in images endpoint [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/347 (https://phabricator.wikimedia.org/T431679)
[17:09:16] <wikibugs>	 (03open) 10dcaro: package: move to poetry base [toolforge-repos/k8s-status] - 10https://gitlab.wikimedia.org/toolforge-repos/k8s-status/-/merge_requests/9
[17:10:30] <wikibugs>	 (03update) 10dcaro: package: move to poetry base [toolforge-repos/k8s-status] - 10https://gitlab.wikimedia.org/toolforge-repos/k8s-status/-/merge_requests/9
[17:12:46] <wikibugs>	 (03open) 10dcaro: package: move to poetry base [toolforge-repos/k8s-status] - 10https://gitlab.wikimedia.org/toolforge-repos/k8s-status/-/merge_requests/10
[17:12:56] <wikibugs>	 (03close) 10dcaro: package: move to poetry base [toolforge-repos/k8s-status] - 10https://gitlab.wikimedia.org/toolforge-repos/k8s-status/-/merge_requests/9
[17:13:16] <wikibugs>	 (03update) 10dcaro: ci: use trixie [toolforge-repos/k8s-status] (move_to_poetry) - 10https://gitlab.wikimedia.org/toolforge-repos/k8s-status/-/merge_requests/8
[17:13:32] <wikibugs>	 (03update) 10dcaro: ci: use trixie [toolforge-repos/k8s-status] (move_to_poetry) - 10https://gitlab.wikimedia.org/toolforge-repos/k8s-status/-/merge_requests/8
[17:23:53] <wikibugs>	 10Tool-wikicrawler: Wiki Crawler: Interactive visual article structure explorer on Toolforge - https://phabricator.wikimedia.org/T440380 (10Adityakumar0545) 03NEW
[17:23:58] <wikibugs>	 (03update) 10dcaro: package: move to poetry base [toolforge-repos/k8s-status] - 10https://gitlab.wikimedia.org/toolforge-repos/k8s-status/-/merge_requests/10
[17:25:41] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
[17:26:02] <wikibugs>	 (03update) 10dcaro: package: move to poetry base [toolforge-repos/k8s-status] - 10https://gitlab.wikimedia.org/toolforge-repos/k8s-status/-/merge_requests/10
[17:29:09] <wikibugs>	 (03update) 10dcaro: package: move to poetry base [toolforge-repos/k8s-status] - 10https://gitlab.wikimedia.org/toolforge-repos/k8s-status/-/merge_requests/10
[17:29:27] <wikibugs>	 (03approved) 10dcaro: maintain-kubeusers: bump to 0.0.203-20261005123458-89615cdc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1452 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[17:29:31] <wikibugs>	 (03merge) 10dcaro: maintain-kubeusers: bump to 0.0.203-20261005123458-89615cdc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1452 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[17:30:53] <wikibugs>	 (03update) 10dcaro: package: move to poetry base [toolforge-repos/k8s-status] - 10https://gitlab.wikimedia.org/toolforge-repos/k8s-status/-/merge_requests/10
[17:32:24] <wikibugs>	 (03update) 10dcaro: package: move to poetry base [toolforge-repos/k8s-status] - 10https://gitlab.wikimedia.org/toolforge-repos/k8s-status/-/merge_requests/10
[17:33:12] <wmcs-alerts>	 FIRING: ToolFrequentRestarts: Containers of k8s-status are restarting repeatedly - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolFrequentRestarts  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolFrequentRestarts
[17:33:50] <wikibugs>	 (03update) 10dcaro: package: move to poetry base [toolforge-repos/k8s-status] - 10https://gitlab.wikimedia.org/toolforge-repos/k8s-status/-/merge_requests/10
[17:40:58] <wikibugs>	 (03update) 10dcaro: package: move to poetry base [toolforge-repos/k8s-status] - 10https://gitlab.wikimedia.org/toolforge-repos/k8s-status/-/merge_requests/10
[17:45:37] <wikibugs>	 (03update) 10dcaro: package: move to poetry base [toolforge-repos/k8s-status] - 10https://gitlab.wikimedia.org/toolforge-repos/k8s-status/-/merge_requests/10
[17:47:17] <wmcs-alerts>	 FIRING: ToolWebserviceErrors: The web service of k8s-status is returning errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolWebserviceErrors  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolWebserviceErrors
[17:57:17] <wmcs-alerts>	 RESOLVED: ToolWebserviceErrors: The web service of k8s-status is returning errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolWebserviceErrors  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolWebserviceErrors
[18:08:59] <wmcs-alerts>	 FIRING: QuarryDown: Quarry application is unreachable   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown
[18:09:03] <wmcs-alerts>	 FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown
[18:13:59] <wmcs-alerts>	 RESOLVED: QuarryDown: Quarry application is unreachable   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown
[18:14:03] <wmcs-alerts>	 RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown
[18:43:12] <wmcs-alerts>	 RESOLVED: ToolFrequentRestarts: Containers of k8s-status are restarting repeatedly - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolFrequentRestarts  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolFrequentRestarts
[18:44:42] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Upgrade (replace) beta|deployment-prep event platform VPS instances - https://phabricator.wikimedia.org/T429497#12403144 (10tchin) All instances have been migrated now
[18:45:58] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Upgrade (replace) beta|deployment-prep event platform VPS instances - https://phabricator.wikimedia.org/T429497#12403146 (10Ottomata) 🎉
[18:52:17] <wikibugs>	 10Tool-wikicrawler: Wiki Crawler: Interactive visual article structure explorer on Toolforge - https://phabricator.wikimedia.org/T440380#12403181 (10Adityakumar0545)
[19:51:21] <wikibugs>	 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: jobs-api: Expose state and webservice_defaults on /images endpoint - https://phabricator.wikimedia.org/T431679#12403392 (10Raymond_Ndibe) >>! In T431679#12167625, @dcaro wrote: > @Raymond_Ndibe If we go with not using jobs-api directly,...
[20:51:59] <wikibugs>	 10Cloud-VPS, 06tools-infrastructure-team: Consider updating the default CPU featureset for nova VMs - https://phabricator.wikimedia.org/T440400 (10Andrew) 03NEW
[21:03:06] <wikibugs>	 10Cloud-VPS, 06tools-infrastructure-team: Consider updating the default CPU featureset for nova VMs - https://phabricator.wikimedia.org/T440400#12403708 (10Andrew)
[21:27:16] <wikibugs>	 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Consider updating the default CPU model for nova VMs - https://phabricator.wikimedia.org/T440400#12403827 (10Andrew)
[23:34:57] <wikibugs>	 10VPS-project-Wikistats: Add magwiki to wikistats - https://phabricator.wikimedia.org/T428284#12404268 (10Dzahn) a:03Dzahn
[23:44:27] <wikibugs>	 10VPS-project-Wikistats: Add magwiki to wikistats - https://phabricator.wikimedia.org/T428284#12404280 (10Dzahn) 05Open→03Resolved ` dzahn@wikistats-trixie:~$   sudo -u wikistatsuser /usr/bin/php /usr/lib/wikistats/update.php wp prefix mag sent query: 'select * from wikipedias where prefix="mag"'. A(1/1)...
[23:47:10] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 10VPS-project-Wikistats: cloud VPS - puppet broken - cinderutils 'fstype must not be an empty string' - https://phabricator.wikimedia.org/T410936#12404283 (10Dzahn) 05Open→03Resolved a:03Dzahn ` Since the bookworm -> trixie migratio...