[00:42:43] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate maps-osmdb from bullseye to .. - https://phabricator.wikimedia.org/T432955#12235018 (10saper) 05Open→03In progress p:05Triage→03Medium a:03saper [03:07:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:08:56] FIRING: CloudVPSDesignateLeaks: Detected 8 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:12:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:13:07] 10Tool-wikinewsie: Wikinewsie cards to be sorted by type based on Wikidata - https://phabricator.wikimedia.org/T434836#12235120 (10Habst) 05In progress→03Resolved [03:13:45] 10Tool-wikinewsie: Wikinewsie cards to be sorted by type based on Wikidata - https://phabricator.wikimedia.org/T434836#12235121 (10Habst) Done, see Event/Person/etc chips based on Wikidata on the main page [03:41:18] (03update) 10raymond-ndibe: refactor: move observability fields out of CommonOptions into ObservabilityOptions [repos/cloud/toolforge/jobs-api] (rename_common_job_to_common_options) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/363 (https://phabricator.wikimedia.org/T434166) [03:41:28] (03update) 10raymond-ndibe: models: support webservice job-type [repos/cloud/toolforge/jobs-api] (models_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) [04:14:35] (03update) 10raymond-ndibe: refactor: move observability fields out of CommonOptions into ObservabilityOptions [repos/cloud/toolforge/jobs-api] (rename_common_job_to_common_options) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/363 (https://phabricator.wikimedia.org/T434166) [04:14:40] (03update) 10raymond-ndibe: refactor: move observability fields out of CommonOptions into ObservabilityOptions [repos/cloud/toolforge/jobs-api] (rename_common_job_to_common_options) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/363 (https://phabricator.wikimedia.org/T434166) [04:17:11] (03update) 10raymond-ndibe: test continuous job webservice [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1131 (https://phabricator.wikimedia.org/T348755) [04:20:46] 06cloud-services-team, 10PAWS, 06tools-platform-team: PAWS: restore nbgitpuller — removing it was never necessary - https://phabricator.wikimedia.org/T434973#12235158 (10Audiodude) I appreciate the context. I actually already have the one line PR ready, just need someone to merge: https://github.com/toolforg... [04:29:01] (03approved) 10raymond-ndibe: gen: update toolforge models [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/183 (owner: 10dcaro) [04:29:03] (03update) 10raymond-ndibe: gen: update toolforge models [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/183 (owner: 10dcaro) [04:29:39] (03update) 10raymond-ndibe: models: move cmd field out of CommonJob into job-type models [repos/cloud/toolforge/jobs-api] (increase_test_coverage) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/364 (https://phabricator.wikimedia.org/T434166) [04:34:27] (03approved) 10raymond-ndibe: core.models: declare the models not accepting extras [repos/cloud/toolforge/jobs-api] (rename_common_job_to_common_options) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/369 (https://phabricator.wikimedia.org/T434166) (owner: 10dcaro) [05:15:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:25:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:27:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:32:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:37:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:32:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:37:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:42:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:50:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:55:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:00:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:03:18] (03open) 10pacmand: Add optional image pre-processing to suppress show-through [toolforge-repos/ocr] - 10https://gitlab.wikimedia.org/toolforge-repos/ocr/-/merge_requests/14 (https://phabricator.wikimedia.org/T435428) [07:08:56] FIRING: CloudVPSDesignateLeaks: Detected 8 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:40:23] (03update) 10mahveotm: cancel: split latest deployment endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/182 (https://phabricator.wikimedia.org/T434499) [07:50:03] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance proxy-5 in project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [08:04:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:06:50] 10Cloud-VPS, 06tools-infrastructure-team: Decommission "onfire" Cloud VPS project - https://phabricator.wikimedia.org/T435387#12235414 (10taavi) a:03taavi [08:07:01] 10Cloud-VPS, 06tools-infrastructure-team: Decommission "onfire" Cloud VPS project - https://phabricator.wikimedia.org/T435387#12235416 (10taavi) 05Open→03Resolved project deleted. [08:09:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:10:49] 10Cloud-VPS, 06tools-infrastructure-team: Request for object_storage role - https://phabricator.wikimedia.org/T435414#12235427 (10taavi) 05Open→03Resolved a:03taavi `lang=shell-session taavi@cloudcontrol1011 ~ $ os role add --user "Peony Server" --project globaleducation object_storage ` Done! [08:11:59] 10Tool-inteGraality, 10Toolhub: Attempting to deduplicate integraality Toolhub records resulted in lost annotations, list removal, and the expected crawler result being ignored - https://phabricator.wikimedia.org/T435159#12235435 (10JeanFred) (One thing to note: the new toolinfo.json was not valid at first, I... [08:14:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:32:07] 10Tools, 07I18n: Add i18n support to Copyvio Detector [AOI] - https://phabricator.wikimedia.org/T110124#12235524 (10Nemoralis) 05Open→03Resolved a:03Nemoralis https://github.com/earwig/copyvios/pull/68 [08:33:39] 10Toolforge, 06tools-platform-team, 07OKR-Work: Email tool maintainers when an alert condition is detected - https://phabricator.wikimedia.org/T432865#12235543 (10fgiunchedi) Option A seems best to me: very little implementation time/effort compared to the alternatives and the external "surface" for alertman... [08:41:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:44:07] (03open) 10dcaro: fix fake types [repos/cloud/toolforge/jobs-api] (rename_common_job_to_common_options) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/370 [08:44:54] (03update) 10dcaro: tests: fix fake return types [repos/cloud/toolforge/jobs-api] (increase_test_coverage) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/370 [08:46:15] (03update) 10dcaro: tests: fix fake return types [repos/cloud/toolforge/jobs-api] (increase_test_coverage) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/370 [08:46:23] (03update) 10dcaro: refactor: move observability fields out of CommonOptions into ObservabilityOptions [repos/cloud/toolforge/jobs-api] (dont_allow_extras_by_default) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/363 (https://phabricator.wikimedia.org/T434166) (owner: 10raymond-ndibe) [08:46:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:47:13] (03update) 10dcaro: tests: fix fake return types [repos/cloud/toolforge/jobs-api] (dont_allow_extras_by_default) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/370 [08:49:33] (03update) 10dcaro: refactor: move observability fields out of CommonOptions into ObservabilityOptions [repos/cloud/toolforge/jobs-api] (dont_allow_extras_by_default) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/363 (https://phabricator.wikimedia.org/T434166) (owner: 10raymond-ndibe) [08:49:51] (03update) 10dcaro: refactor: move observability fields out of CommonOptions into ObservabilityOptions [repos/cloud/toolforge/jobs-api] (fix_fake_types) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/363 (https://phabricator.wikimedia.org/T434166) (owner: 10raymond-ndibe) [08:59:59] 10Toolforge, 06tools-infrastructure-team: See if we can borrow parts of the wikiprod WAF for Toolforge - https://phabricator.wikimedia.org/T409759#12235684 (10fnegri) One example of a tool negatively affected by scraping: https://github.com/WDscholia/scholia/issues/2794 [09:18:43] (03open) 10dcaro: core.models: make get_resolved_job not have side effects [repos/cloud/toolforge/jobs-api] (models_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/371 [09:18:54] (03update) 10dcaro: core.models: make get_resolved_job not have side effects [repos/cloud/toolforge/jobs-api] (models_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/371 [09:19:36] (03update) 10dcaro: core.models: make get_resolved_job not have side effects [repos/cloud/toolforge/jobs-api] (models_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/371 [09:37:12] (03update) 10dcaro: core.models: make get_resolved_job not have side effects [repos/cloud/toolforge/jobs-api] (models_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/371 [09:40:33] 10Tool-itwiki: BotCancellazioni: rollback "cllimit=500" workaround after T433922 has been resolved - https://phabricator.wikimedia.org/T434598#12235899 (10valerio.bozzolan) >>! In T434598#12227903, @Umherirrender wrote: > Setting limits is up to the client calling the api. My friend, I'm failing in understandin... [09:41:10] (03update) 10dcaro: core.models: make get_resolved_job not have side effects [repos/cloud/toolforge/jobs-api] (models_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/371 [09:43:28] (03update) 10dcaro: tests: fix fake return types [repos/cloud/toolforge/jobs-api] (dont_allow_extras_by_default) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/370 [09:43:48] 10Tool-itwiki: BotCancellazioni: rollback "cllimit=500" workaround after T433922 has been resolved - https://phabricator.wikimedia.org/T434598#12235924 (10valerio.bozzolan) I mean, I was* going to rollback. Now I'm stalled trying to understand if you want to keep it here and why. As again, I shared my personal... [09:44:41] (03merge) 10dcaro: tests: add behavior tests for previously untested code branches [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/362 (https://phabricator.wikimedia.org/T434166) (owner: 10raymond-ndibe) [09:44:47] (03update) 10dcaro: models: move cmd field out of CommonJob into job-type models [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/364 (https://phabricator.wikimedia.org/T434166) (owner: 10raymond-ndibe) [09:48:17] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.560-20260820094456-cdc612ef [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1383 (https://phabricator.wikimedia.org/T434166) [09:48:21] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.560-20260820094456-cdc612ef [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1383 (https://phabricator.wikimedia.org/T434166) [09:49:26] (03approved) 10dcaro: models: move cmd field out of CommonJob into job-type models [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/364 (https://phabricator.wikimedia.org/T434166) (owner: 10raymond-ndibe) [10:00:04] (03update) 10dcaro: refactor: rename CommonJob to CommonOptions [repos/cloud/toolforge/jobs-api] (move_cmd_out_of_common_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/367 (https://phabricator.wikimedia.org/T434166) (owner: 10raymond-ndibe) [10:01:41] (03merge) 10dcaro: models: move cmd field out of CommonJob into job-type models [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/364 (https://phabricator.wikimedia.org/T434166) (owner: 10raymond-ndibe) [10:01:46] (03update) 10dcaro: refactor: rename CommonJob to CommonOptions [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/367 (https://phabricator.wikimedia.org/T434166) (owner: 10raymond-ndibe) [10:04:56] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.561-20260820100155-7d89ce25 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1383 (https://phabricator.wikimedia.org/T434166) [10:05:03] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.561-20260820100155-7d89ce25 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1383 (https://phabricator.wikimedia.org/T434166) [10:06:00] (03merge) 10dcaro: refactor: rename CommonJob to CommonOptions [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/367 (https://phabricator.wikimedia.org/T434166) (owner: 10raymond-ndibe) [10:06:01] (03update) 10dcaro: core.models: declare the models not accepting extras [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/369 (https://phabricator.wikimedia.org/T434166) [10:06:14] (03update) 10dcaro: core.models: declare the models not accepting extras [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/369 (https://phabricator.wikimedia.org/T434166) [10:06:50] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [10:06:50] !log dcaro@cloudcumin1001 toolsbeta END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api [10:09:25] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.562-20260820100610-60a21139 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1383 (https://phabricator.wikimedia.org/T434166) [10:09:29] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.562-20260820100610-60a21139 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1383 (https://phabricator.wikimedia.org/T434166) [10:09:53] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [10:13:08] (03approved) 10dcaro: toolforge-cd: generate deployment description from Git [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/97 (https://phabricator.wikimedia.org/T393169 https://phabricator.wikimedia.org/T401993) (owner: 10lucaswerkmeister) [10:13:24] (03merge) 10dcaro: toolforge-cd: generate deployment description from Git [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/97 (https://phabricator.wikimedia.org/T393169 https://phabricator.wikimedia.org/T401993) (owner: 10lucaswerkmeister) [10:13:49] FIRING: PuppetDisabled: Puppet disabled on cloudcephosd2007-dev:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=wmcs&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [10:23:12] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [10:30:45] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [10:42:09] (03update) 10dcaro: refactor: move fields out of CommonOptions into *Options reusable classes [repos/cloud/toolforge/jobs-api] (fix_fake_types) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/363 (https://phabricator.wikimedia.org/T434166) (owner: 10raymond-ndibe) [10:44:16] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [10:47:50] (03approved) 10dcaro: jobs-api: bump to 0.0.562-20260820100610-60a21139 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1383 (https://phabricator.wikimedia.org/T434166) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:47:55] (03merge) 10dcaro: jobs-api: bump to 0.0.562-20260820100610-60a21139 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1383 (https://phabricator.wikimedia.org/T434166) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:48:24] (03merge) 10dcaro: core.models: declare the models not accepting extras [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/369 (https://phabricator.wikimedia.org/T434166) [10:48:25] (03update) 10dcaro: tests: fix fake return types [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/370 [10:51:19] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.563-20260820104835-42beb4f6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1384 (https://phabricator.wikimedia.org/T434166) [10:51:29] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.563-20260820104835-42beb4f6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1384 (https://phabricator.wikimedia.org/T434166) [10:51:50] (03update) 10dcaro: tests: fix fake return types [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/370 [10:52:11] (03approved) 10dcaro: tests: fix fake return types [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/370 [10:54:18] (03merge) 10dcaro: tests: fix fake return types [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/370 [10:54:22] (03update) 10dcaro: refactor: move fields out of CommonOptions into *Options reusable classes [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/363 (https://phabricator.wikimedia.org/T434166) (owner: 10raymond-ndibe) [10:57:27] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.564-20260820105430-291142bc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1384 (https://phabricator.wikimedia.org/T434166) [10:57:37] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.564-20260820105430-291142bc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1384 (https://phabricator.wikimedia.org/T434166) [11:06:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:08:56] FIRING: CloudVPSDesignateLeaks: Detected 8 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:11:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:16:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:22:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:26:24] 10Tool-itwiki: BotCancellazioni: rollback "cllimit=500" workaround after T433922 has been resolved - https://phabricator.wikimedia.org/T434598#12236327 (10valerio.bozzolan) 05Stalled→03Open [11:27:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:43:18] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [11:43:58] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [11:45:39] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [11:53:32] 10Tool-wikimedia-attribution, 06MediaWiki-API-Platform-Team, 10MediaWiki-REST-API, 06MediaWiki-Core-Platform-Team (Kanban), and 2 others: Download CTA should be present only on Wikipedia projects - https://phabricator.wikimedia.org/T434214#12236446 (10pmiazga) [11:58:14] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [12:05:00] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [12:05:05] (03open) 10dcaro: tests: add some tests for runtime.build_start [repos/cloud/toolforge/components-api] (add_build_envvars) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/185 (https://phabricator.wikimedia.org/T434496) [12:05:34] (03update) 10dcaro: api: add envvars to source build [repos/cloud/toolforge/components-api] (update_toolforge_models) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/184 (https://phabricator.wikimedia.org/T434496) [12:06:00] (03update) 10dcaro: tests: add some tests for runtime.build_start [repos/cloud/toolforge/components-api] (add_build_envvars) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/185 (https://phabricator.wikimedia.org/T434496) [12:06:30] (03update) 10dcaro: tests: add some tests for runtime.build_start [repos/cloud/toolforge/components-api] (add_build_envvars) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/185 (https://phabricator.wikimedia.org/T434496) [12:09:10] (03update) 10dcaro: tests: add some tests for runtime.build_start [repos/cloud/toolforge/components-api] (add_build_envvars) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/185 (https://phabricator.wikimedia.org/T434496) [12:09:38] (03update) 10dcaro: tests: add some tests for runtime.build_start [repos/cloud/toolforge/components-api] (add_build_envvars) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/185 (https://phabricator.wikimedia.org/T434496) [12:11:13] (03update) 10dcaro: tests: add some tests for runtime.build_start [repos/cloud/toolforge/components-api] (add_build_envvars) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/185 (https://phabricator.wikimedia.org/T434496) [12:12:41] (03merge) 10dcaro: gen: update toolforge models [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/183 [12:12:48] (03update) 10dcaro: api: add envvars to source build [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/184 (https://phabricator.wikimedia.org/T434496) [12:12:59] (03update) 10dcaro: cancel: split latest deployment endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/182 (https://phabricator.wikimedia.org/T434499) (owner: 10mahveotm) [12:12:59] (03update) 10dcaro: cancel: split latest deployment endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/182 (https://phabricator.wikimedia.org/T434499) (owner: 10mahveotm) [12:13:06] (03update) 10dcaro: cancel: split latest deployment endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/182 (https://phabricator.wikimedia.org/T434499) (owner: 10mahveotm) [12:15:02] 06cloud-services-team, 10PAWS, 06tools-platform-team: PAWS: restore nbgitpuller — removing it was never necessary - https://phabricator.wikimedia.org/T434973#12236568 (10fnegri) 05Open→03Resolved a:03fnegri @Audiodude I added that comment yesterday while quickly triaging a bunch of new tasks. You c... [12:15:32] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.208-20260820121257-28ffdc12 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1385 [12:16:00] (03update) 10dcaro: update handler to return empty list when no deployments [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/174 (https://phabricator.wikimedia.org/T429229) (owner: 10buxtang) [12:17:19] (03update) 10dcaro: update handler to return empty list when no deployments [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/174 (https://phabricator.wikimedia.org/T429229) (owner: 10buxtang) [12:18:14] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [12:20:00] (03approved) 10dcaro: jobs-api: bump to 0.0.564-20260820105430-291142bc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1384 (https://phabricator.wikimedia.org/T434166) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:20:05] (03merge) 10dcaro: jobs-api: bump to 0.0.564-20260820105430-291142bc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1384 (https://phabricator.wikimedia.org/T434166) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:33:06] (03update) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/75 [12:38:09] 10Toolforge, 06tools-infrastructure-team: See if we can borrow parts of the wikiprod WAF for Toolforge - https://phabricator.wikimedia.org/T409759#12236790 (10Lferreira) Another tool negatively affected by scrappers https://phabricator.wikimedia.org/T432878 [12:47:22] (03update) 10dcaro: update handler to return empty list when no deployments [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/174 (https://phabricator.wikimedia.org/T429229) (owner: 10buxtang) [12:47:25] (03update) 10dcaro: update handler to return empty list when no deployments [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/174 (https://phabricator.wikimedia.org/T429229) (owner: 10buxtang) [12:48:23] (03update) 10dcaro: update handler to return empty list when no deployments [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/174 (https://phabricator.wikimedia.org/T429229) (owner: 10buxtang) [12:56:08] 10Toolforge, 06tools-platform-team: [toolsdb] 2026-08-18 Transaction History Length growing too much - https://phabricator.wikimedia.org/T435220#12236848 (10fnegri) The impact on the ToolsDB disk metrics is also very visible when `rustbot` is running (from 2026-08-15 to 2026-08-19): {F99229323} [12:58:40] 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: [infra] Allow some opt-in scraping solution for tools - https://phabricator.wikimedia.org/T433395#12236893 (10fnegri) [12:58:42] 10Toolforge, 06tools-platform-team, 07good first task: [builds-cli] does not output valid json when there's no builds - https://phabricator.wikimedia.org/T429229#12236892 (10dcaro) I'll close this and open a new task for the components-api/cli [12:58:48] 06cloud-services-team, 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: Block web crawlers from accessing Cloud Services - https://phabricator.wikimedia.org/T226688#12236894 (10fnegri) [12:59:01] 10Toolforge, 06tools-platform-team, 07good first task: [builds-cli] does not output valid json when there's no builds - https://phabricator.wikimedia.org/T429229#12236895 (10dcaro) 05In progress→03Resolved [13:00:05] 06tools-platform-team: [components-api,components-cli] don't error when there's no deployments - https://phabricator.wikimedia.org/T435492 (10dcaro) 03NEW [13:00:11] 06tools-platform-team: [components-api,components-cli] don't error when there's no deployments - https://phabricator.wikimedia.org/T435492#12236911 (10dcaro) p:05Triage→03Low [13:00:42] (03update) 10raymond-ndibe: test continuous job webservice [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1131 (https://phabricator.wikimedia.org/T348755) [13:01:08] 06tools-platform-team: [components-api,components-cli] don't error when there's no deployments - https://phabricator.wikimedia.org/T435492#12236913 (10dcaro) a:03Buxtang It started already on {T429229} [13:01:21] (03update) 10dcaro: update handler to return empty list when no deployments [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/174 (https://phabricator.wikimedia.org/T435492) (owner: 10buxtang) [13:01:37] 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: [infra] Allow some opt-in scraping solution for tools - https://phabricator.wikimedia.org/T433395#12236918 (10fnegri) Should the title be "Allow some opt-in ANTI-scraping solution for tools"? Can you also add some more design details in the tas... [13:04:16] (03update) 10raymond-ndibe: models: support webservice job-type [repos/cloud/toolforge/jobs-api] (models_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) [13:04:41] (03update) 10dcaro: refactor: move fields out of CommonOptions into *Options reusable classes [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/363 (https://phabricator.wikimedia.org/T434166) (owner: 10raymond-ndibe) [13:04:59] (03update) 10dcaro: core.models: make get_resolved_job not have side effects [repos/cloud/toolforge/jobs-api] (models_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/371 [13:14:06] (03update) 10raymond-ndibe: core.models: make get_resolved_job not have side effects [repos/cloud/toolforge/jobs-api] (models_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/371 (owner: 10dcaro) [13:14:07] (03approved) 10raymond-ndibe: core.models: make get_resolved_job not have side effects [repos/cloud/toolforge/jobs-api] (models_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/371 (owner: 10dcaro) [13:14:23] (03merge) 10raymond-ndibe: core.models: make get_resolved_job not have side effects [repos/cloud/toolforge/jobs-api] (models_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/371 (owner: 10dcaro) [13:14:25] (03update) 10raymond-ndibe: refactor: move fields out of CommonOptions into *Options reusable classes [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/363 (https://phabricator.wikimedia.org/T434166) [13:18:42] RESOLVED: CloudVPSDesignateLeaks: Detected 8 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:20:34] (03approved) 10dcaro: update handler to return empty list when no deployments [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/174 (https://phabricator.wikimedia.org/T435492) (owner: 10buxtang) [13:21:01] (03update) 10dcaro: components-api: bump to 0.0.208-20260820121257-28ffdc12 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1385 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:21:12] (03update) 10raymond-ndibe: refactor: move fields out of CommonOptions into *Options reusable classes [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/363 (https://phabricator.wikimedia.org/T434166) [13:21:17] (03merge) 10dcaro: update handler to return empty list when no deployments [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/174 (https://phabricator.wikimedia.org/T435492) (owner: 10buxtang) [13:23:51] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.209-20260820132129-dac7467a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1385 [13:24:02] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.209-20260820132129-dac7467a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1385 [13:24:40] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [13:24:46] (03open) 10dcaro: list: show helpful message when there's no deployments [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/94 (https://phabricator.wikimedia.org/T435492) [13:26:59] (03update) 10dcaro: list: show helpful message when there's no deployments [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/94 (https://phabricator.wikimedia.org/T435492) [13:28:53] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [13:29:37] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [13:31:27] (03update) 10dcaro: list: show helpful message when there's no deployments [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/94 (https://phabricator.wikimedia.org/T435492) [13:33:30] PROBLEM - mysqld processes on clouddb1025 is CRITICAL: PROCS CRITICAL: 1 process with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [13:34:37] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [13:34:39] (03update) 10dcaro: list: show helpful message when there's no deployments [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/94 (https://phabricator.wikimedia.org/T435492) [13:36:25] FIRING: SystemdUnitFailed: wmf-pt-kill@s6.service on clouddb1025:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:37:30] RECOVERY - mysqld processes on clouddb1025 is OK: PROCS OK: 1 process with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [13:38:21] (03merge) 10dcaro: components-api: bump to 0.0.209-20260820132129-dac7467a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1385 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:44:23] (03update) 10dcaro: list: show helpful message when there's no deployments [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/94 (https://phabricator.wikimedia.org/T435492) [13:45:10] 10Data-Services, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Remove hdfs fuse mounts from clouddumps - https://phabricator.wikimedia.org/T434794#12237135 (10BTullis) [13:46:39] (03update) 10raymond-ndibe: models: support webservice job-type [repos/cloud/toolforge/jobs-api] (models_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) [13:47:47] (03update) 10dcaro: list: show helpful message when there's no deployments [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/94 (https://phabricator.wikimedia.org/T435492) [13:49:24] (03update) 10dcaro: list: show helpful message when there's no deployments [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/94 (https://phabricator.wikimedia.org/T435492) [13:52:12] (03update) 10raymond-ndibe: refactor: move fields out of CommonOptions into *Options reusable classes [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/363 (https://phabricator.wikimedia.org/T434166) [13:55:07] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#12237187 (10BTullis) a:03BTullis I'll take this task for now, although it may be someone else from #data-... [13:55:50] (03update) 10dcaro: refactor: move fields out of CommonOptions into *Options reusable classes [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/363 (https://phabricator.wikimedia.org/T434166) (owner: 10raymond-ndibe) [13:55:55] (03approved) 10dcaro: refactor: move fields out of CommonOptions into *Options reusable classes [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/363 (https://phabricator.wikimedia.org/T434166) (owner: 10raymond-ndibe) [13:57:38] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387#12237196 (10Andrew) I'm draining and destroying three servers in order to run fio tests (and because at some point DELL th... [13:58:09] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) [13:58:17] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [13:58:46] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387#12237201 (10Andrew) Also, some HW notes on https://docs.google.com/spreadsheets/d/1YdP6qkH2shhdDTJ96kWU7VbSZ197likPvZtRsEf... [14:00:50] (03open) 10dcaro: poetry: add the missing dev dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/372 [14:00:55] (03update) 10raymond-ndibe: models: support webservice job-type [repos/cloud/toolforge/jobs-api] (models_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) [14:02:24] (03update) 10dcaro: list: show helpful message when there's no deployments [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/94 (https://phabricator.wikimedia.org/T435492) [14:04:59] 10Data-Services, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-08-07 - 2026-08-28), 13Patch-For-Review: Remove hdfs fuse mounts from clouddumps - https://phabricator.wikimedia.org/T434794#12237232 (10taavi) 05Open→03Resolved a:03taavi The FUSE mounts have been removed. The `hdfs_rsync_*`... [14:06:41] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12237244 (10fnegri) I removed s6 from clouddb1025, with the following steps: * depooled clouddb1025 in confctl (bo... [14:08:10] (03update) 10dcaro: poetry: add the missing dev dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/372 [14:08:36] (03approved) 10raymond-ndibe: poetry: add the missing dev dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/372 (owner: 10dcaro) [14:10:26] (03update) 10raymond-ndibe: list: show helpful message when there's no deployments [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/94 (https://phabricator.wikimedia.org/T435492) (owner: 10dcaro) [14:10:27] (03approved) 10raymond-ndibe: list: show helpful message when there's no deployments [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/94 (https://phabricator.wikimedia.org/T435492) (owner: 10dcaro) [14:11:26] (03merge) 10dcaro: poetry: add the missing dev dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/372 [14:11:33] (03merge) 10dcaro: list: show helpful message when there's no deployments [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/94 (https://phabricator.wikimedia.org/T435492) [14:12:49] (03update) 10raymond-ndibe: refactor: move fields out of CommonOptions into *Options reusable classes [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/363 (https://phabricator.wikimedia.org/T434166) [14:13:51] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-cli [14:13:51] 06cloud-services-team, 10PAWS, 06tools-platform-team: PAWS: restore nbgitpuller — removing it was never necessary - https://phabricator.wikimedia.org/T434973#12237286 (10Audiodude) Thanks, I really appreciate it! [14:13:59] (03open) 10dcaro: d/changelog: bump to 0.0.20 [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/95 (https://phabricator.wikimedia.org/T435492) [14:14:03] FIRING: PuppetDisabled: Puppet disabled on cloudcephosd2007-dev:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=wmcs&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [14:15:28] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.565-20260820141147-c8efe18e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1386 [14:15:35] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.565-20260820141147-c8efe18e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1386 [14:18:02] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli [14:18:18] (03update) 10raymond-ndibe: refactor: move fields out of CommonOptions into *Options reusable classes [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/363 (https://phabricator.wikimedia.org/T434166) [14:19:42] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team, and 3 others: Move WMCS off of Icinga and introduce alertmanager - https://phabricator.wikimedia.org/T328502#12237329 (10Andrew) >>! In T328502#12231204, @fgiunchedi wrote: > To be honest since we're already checking dns from met... [14:21:31] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-cli [14:23:49] RESOLVED: PuppetDisabled: Puppet disabled on cloudcephosd2007-dev:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=wmcs&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [14:24:46] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli [14:25:24] (03approved) 10dcaro: d/changelog: bump to 0.0.20 [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/95 (https://phabricator.wikimedia.org/T435492) [14:25:29] (03merge) 10dcaro: d/changelog: bump to 0.0.20 [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/95 (https://phabricator.wikimedia.org/T435492) [14:26:22] 10Tool-linkdata: Homepage with docs - https://phabricator.wikimedia.org/T435501 (10Audiodude) 03NEW [14:26:43] 10Tool-linkdata: Homepage with docs - https://phabricator.wikimedia.org/T435501#12237380 (10Audiodude) a:03Audiodude [14:26:47] 06tools-platform-team, 13Patch-For-Review: [components-api,components-cli] don't error when there's no deployments - https://phabricator.wikimedia.org/T435492#12237381 (10dcaro) Released the cli side too \o/, thanks @Buxtang [14:26:56] 06tools-platform-team, 13Patch-For-Review: [components-api,components-cli] don't error when there's no deployments - https://phabricator.wikimedia.org/T435492#12237384 (10dcaro) 05Open→03Resolved [14:28:34] (03update) 10raymond-ndibe: refactor: move fields out of CommonOptions into *Options reusable classes [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/363 (https://phabricator.wikimedia.org/T434166) [14:29:23] 10PAWS, 06tools-infrastructure-team, 06tools-platform-team: Make sure dumps-nfs mounts are propagated inside PAWS containers - https://phabricator.wikimedia.org/T432583#12237397 (10fgiunchedi) After a couple of false starts, I have deployed the changes to paws so notebooks now have: ` dumps-nfs.wikimedia.or... [14:32:49] 10Data-Services, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Remove hdfs fuse mounts from clouddumps - https://phabricator.wikimedia.org/T434794#12237421 (10BTullis) Great, thanks @taavi - I hadn't spotted that everything was already using `hdfs-rsync` With regard to this... [14:39:22] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#12237470 (10BTullis) >>! In T403154#12237187, @BTullis wrote: > An upgrade to trixie isn't possible at the... [14:44:00] (03PS1) 10Urbanecm: cswiki/mostLinkedDisambigs: Use page_props rather than category [labs/tools/urbanecmbot] - 10https://gerrit.wikimedia.org/r/1327555 [14:44:00] (03PS1) 10Urbanecm: Fix issues with linktarget [labs/tools/urbanecmbot] - 10https://gerrit.wikimedia.org/r/1327556 [14:45:26] (03CR) 10Urbanecm: [C:03+2] Fix issues with linktarget [labs/tools/urbanecmbot] - 10https://gerrit.wikimedia.org/r/1327556 (owner: 10Urbanecm) [14:45:27] (03CR) 10Urbanecm: [C:03+2] cswiki/mostLinkedDisambigs: Use page_props rather than category [labs/tools/urbanecmbot] - 10https://gerrit.wikimedia.org/r/1327555 (owner: 10Urbanecm) [14:47:50] (03Merged) 10jenkins-bot: cswiki/mostLinkedDisambigs: Use page_props rather than category [labs/tools/urbanecmbot] - 10https://gerrit.wikimedia.org/r/1327555 (owner: 10Urbanecm) [14:47:51] (03Merged) 10jenkins-bot: Fix issues with linktarget [labs/tools/urbanecmbot] - 10https://gerrit.wikimedia.org/r/1327556 (owner: 10Urbanecm) [14:50:43] (03update) 10raymond-ndibe: refactor: move fields out of CommonOptions into *Options reusable classes [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/363 (https://phabricator.wikimedia.org/T434166) [14:50:54] (03approved) 10raymond-ndibe: refactor: move fields out of CommonOptions into *Options reusable classes [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/363 (https://phabricator.wikimedia.org/T434166) [14:51:21] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate maps-osmdb from bullseye to .. - https://phabricator.wikimedia.org/T432955#12237534 (10dschwen) Rebuilding the DB takes a very long time (weeks). So if we could avoid that and do an in-place upgrade that'd be fantastic. [14:53:58] (03open) 10dcaro: toolforge_deploy: reset to FETCH_HEAD when refetching [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/340 [14:55:17] 10Toolforge, 06tools-platform-team, 07OKR-Work: Email tool maintainers when an alert condition is detected - https://phabricator.wikimedia.org/T432865#12237560 (10fnegri) Thanks for the comments. I agree that A looks like a good option and is easier to implement than option B. @dcaro regarding decoupling, I... [15:02:16] 10Toolforge, 06tools-platform-team, 07OKR-Work: Email tool maintainers when an alert condition is detected - https://phabricator.wikimedia.org/T432865#12237588 (10dcaro) > Thanks for the comments. I agree that A looks like a good option and is easier to implement than option B. @dcaro regarding decoupling, I... [15:06:15] (03update) 10dcaro: toolforge_deploy: reset to FETCH_HEAD when refetching [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/340 [15:06:25] (03update) 10dcaro: toolforge_deploy: reset to FETCH_HEAD when refetching [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/340 [15:09:49] (03update) 10pacmand: Add optional image pre-processing to suppress show-through [toolforge-repos/ocr] - 10https://gitlab.wikimedia.org/toolforge-repos/ocr/-/merge_requests/14 (https://phabricator.wikimedia.org/T435428) [15:14:07] (03update) 10dcaro: maintain-harbor: ignore volatile retention schedule time [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1382 (https://phabricator.wikimedia.org/T393878) (owner: 10mahveotm) [15:14:16] (03update) 10dcaro: maintain-harbor: ignore volatile retention schedule time [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1382 (https://phabricator.wikimedia.org/T393878) (owner: 10mahveotm) [15:20:32] (03approved) 10dcaro: maintain-harbor: ignore volatile retention schedule time [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1382 (https://phabricator.wikimedia.org/T393878) (owner: 10mahveotm) [15:21:29] (03merge) 10dcaro: maintain-harbor: ignore volatile retention schedule time [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1382 (https://phabricator.wikimedia.org/T393878) (owner: 10mahveotm) [15:22:08] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [functional-tests] maintain-harbor tests are a bit flaky - https://phabricator.wikimedia.org/T393878#12237663 (10dcaro) 05In progress→03Resolved [15:28:03] FIRING: InstanceDown: Project tools instance tools-puppetserver-01 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [15:30:35] 10Data-Services, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Remove hdfs fuse mounts from clouddumps - https://phabricator.wikimedia.org/T434794#12237714 (10BTullis) [15:36:21] (03update) 10raymond-ndibe: toolforge_deploy: reset to FETCH_HEAD when refetching [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/340 (owner: 10dcaro) [15:36:23] (03approved) 10raymond-ndibe: toolforge_deploy: reset to FETCH_HEAD when refetching [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/340 (owner: 10dcaro) [15:36:53] (03merge) 10raymond-ndibe: refactor: move fields out of CommonOptions into *Options reusable classes [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/363 (https://phabricator.wikimedia.org/T434166) [15:36:57] (03update) 10raymond-ndibe: models: support webservice job-type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) [15:38:03] RESOLVED: InstanceDown: Project tools instance tools-puppetserver-01 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [15:40:29] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.566-20260820153703-a55fc510 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1386 (https://phabricator.wikimedia.org/T434166) [15:40:34] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.566-20260820153703-a55fc510 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1386 (https://phabricator.wikimedia.org/T434166) [15:40:42] (03merge) 10dcaro: toolforge_deploy: reset to FETCH_HEAD when refetching [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/340 [15:41:18] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate maps-osmdb from bullseye to .. - https://phabricator.wikimedia.org/T432955#12237840 (10taavi) >>! In T432955#12237534, @dschwen wrote: > Rebuilding the DB takes a very long time (weeks). So if we could avoid that and do an in-place upgrade that'd be fantastic.... [15:43:17] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [15:56:12] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [15:56:25] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [15:57:18] 10Cloud-VPS (Debian Bullseye Deprecation): Shutdown timeline extension requests for maintained projects - https://phabricator.wikimedia.org/T434103#12237969 (10komla) [16:11:20] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [16:12:26] (03approved) 10raymond-ndibe: jobs-api: bump to 0.0.566-20260820153703-a55fc510 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1386 (https://phabricator.wikimedia.org/T434166) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:12:35] (03merge) 10raymond-ndibe: jobs-api: bump to 0.0.566-20260820153703-a55fc510 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1386 (https://phabricator.wikimedia.org/T434166) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:53:35] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate maps-osmdb from bullseye to .. - https://phabricator.wikimedia.org/T432955#12238287 (10dschwen) >>! In T432955#12237840, @taavi wrote: > Is using [[ https://wikitech.wikimedia.org/wiki/Help:Adding_disk_space_to_Cloud_VPS_instances | a volume ]] to migrate the ex... [16:57:50] !log andrew@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=97) (T429387) [16:57:57] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [17:04:22] (03open) 10tlepage: ping_clinic_duty: add tlepage [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/10 [17:05:56] (03approved) 10dcaro: ping_clinic_duty: add tlepage [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/10 (owner: 10tlepage) [17:07:44] (03merge) 10tlepage: ping_clinic_duty: add tlepage [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/10 [17:17:33] 10Tool-techcontribs: Add summary of on-wiki technical contributions - https://phabricator.wikimedia.org/T411954#12238433 (10Chlod) 05In progress→03Resolved Well, I didn't really follow the API plan in the task description, but the API now exists and the relevant contribution set also now shows up on the... [17:31:13] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) [17:31:14] !log andrew@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=97) (T429387) [17:31:20] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [18:16:25] (03approved) 10pepepiton: links to create items in no-results, results, and works-list pages [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/188 (owner: 10ademola) [18:16:49] (03merge) 10pepepiton: links to create items in no-results, results, and works-list pages [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/188 (owner: 10ademola) [18:49:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:54:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:11:51] 10Tool-itwiki: BotCancellazioni: rollback "cllimit=500" workaround after T433922 has been resolved - https://phabricator.wikimedia.org/T434598#12238965 (10Umherirrender) Personally I am using high limits when using api requests to get most data in one request. I have no idea about the cost on the server side, bu... [19:36:38] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) [19:36:47] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [19:36:56] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12239020 (10VRiley-WMF) I think I figured out what was going on with cloudvirt1054. It doesn't seem to be booting up pro... [19:50:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:55:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:00:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:12:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:17:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:29:36] 10Quarry, 06tools-platform-team: Define a robot policy for Quarry then create robots.txt file - https://phabricator.wikimedia.org/T224338#12239181 (10Framawiki) 05Open→03Resolved a:03Framawiki Was set in the meantime: ` User-Agent: * Disallow: / ` [20:30:22] 10Quarry, 06tools-platform-team, 07good first task: Add nofollow attribute in some links to prevent bots from following unnecessary ones - https://phabricator.wikimedia.org/T197488#12239186 (10Framawiki) Thank you for tagging this task with #good_first_task for Wikimedia newcomers! Newcomers often may not b... [21:17:13] 06cloud-services-team, 10Data-Services, 06tools-platform-team: [wikireplicas] Create views for new wiki minwikiquote - https://phabricator.wikimedia.org/T429946#12239408 (10xcollazo) @fnegri: Sorry for the direct ping, but I saw you took care of {T429938}. Would you be able to help with this ticket, or point... [21:29:14] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate maps-osmdb from bullseye to .. - https://phabricator.wikimedia.org/T432955#12239464 (10saper) Yes, at first we shall stop using the PostgreSQL version as given by Debian. Using PGDG packages it should be possible to move the database volume to PostgreSQL 13 run... [21:36:53] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12239476 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudvirt1... [21:52:51] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12239511 (10VRiley-WMF) [22:10:58] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12239559 (10VRiley-WMF) Okay, so... After troubleshooting it for a while, I thought it may be the CPU that was acting u... [22:15:13] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12239572 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host cloudvirt1054.... [22:42:25] 10Quarry, 06tools-platform-team, 07good first task: Add nofollow attribute in some links to prevent bots from following unnecessary ones - https://phabricator.wikimedia.org/T197488#12239622 (10Osps7) I have submitted a proposed fix for this task: https://github.com/toolforge/quarry/pull/102 [22:49:31] 10Quarry, 06tools-platform-team, 07good first task, 13Patch-For-Review: Add nofollow attribute in some links to prevent bots from following unnecessary ones - https://phabricator.wikimedia.org/T197488#12239655 (10Framawiki) [22:53:15] 10Tool-wikinewsie: Recent Changes page for Wikinewsie - https://phabricator.wikimedia.org/T431876#12239657 (10Habst) 05Open→03In progress a:03Habst [23:39:12] 10Tool-wikinewsie: Recent Changes page for Wikinewsie - https://phabricator.wikimedia.org/T431876#12239758 (10Habst) 05In progress→03Resolved [23:39:50] 10Tool-wikinewsie: Recent Changes page for Wikinewsie - https://phabricator.wikimedia.org/T431876#12239759 (10Habst) Done: https://wikinewsie.toolforge.org/site/changes [23:50:56] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) [23:51:03] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387