[00:07:00] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06Traffic: Replace cache proxies: deployment-cache-(text|upload)08 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436468#12320556 (10BCornwall) a:03BCornwall [00:11:19] 10Tool-wikinewsie: Wikinewsie recent changes page should let you go back further - https://phabricator.wikimedia.org/T437966 (10Pharos) 03NEW [02:00:11] (03PS1) 10MusikAnimal: env: update OPTED_IN projects following WMF-agnostic work [labs/xtools] - 10https://gerrit.wikimedia.org/r/1341482 (https://phabricator.wikimedia.org/T437969) [02:01:40] (03CR) 10MusikAnimal: [C:03+2] env: update OPTED_IN projects following WMF-agnostic work [labs/xtools] - 10https://gerrit.wikimedia.org/r/1341482 (https://phabricator.wikimedia.org/T437969) (owner: 10MusikAnimal) [02:02:49] (03Merged) 10jenkins-bot: env: update OPTED_IN projects following WMF-agnostic work [labs/xtools] - 10https://gerrit.wikimedia.org/r/1341482 (https://phabricator.wikimedia.org/T437969) (owner: 10MusikAnimal) [04:21:54] 10Tool-wikinewsie: Adjustable image alignment on Wikinewsie - https://phabricator.wikimedia.org/T434467#12320795 (10Pharos) It occurs to me that all of the remaining weird images are SVGs with transparent backgrounds. So as a simple step, it would be good if SVGs were treated differently from other image types,... [04:54:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [04:59:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [07:04:59] 10Tool-Pageviews, 07Regression: Massviews Analysis fails due to 414 Request-URI Too Long - https://phabricator.wikimedia.org/T437947#12320933 (10MusikAnimal) 05In progress→03Resolved The extra round trips en masse are costly, but I think this is fixed. I don't see any 414's anymore. For ASCII titles,... [07:17:52] 10Tool-Pageviews, 07I18n, 07Plural-Support: [[Wikimedia:Pageviews-auto-pagepile-error/cs]] needs PLURAL - https://phabricator.wikimedia.org/T199015#12320956 (10MusikAnimal) 05Open→03Invalid Message no longer exists [07:18:31] 10Tool-Pageviews: Auto-pagepile breaks encoding - https://phabricator.wikimedia.org/T161124#12320959 (10MusikAnimal) 05Open→03Invalid The auto-PagePile feature was removed [07:28:05] 10Tool-Pageviews, 10GLAM-Tech: Massviews: bug with pagepile adding extra Wikipedia: on GLAM cultural partnership project pages - https://phabricator.wikimedia.org/T135437#12320989 (10MusikAnimal) 05Open→03Resolved a:03MusikAnimal The bug as written is fixed. It won't error out, though it does query f... [07:30:41] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12320995 (10komla) 05In progress→03Resolved [07:32:29] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12320999 (10komla) [07:33:37] (03CR) 10Volans: [C:03+1] "LGTM" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341302 (https://phabricator.wikimedia.org/T437219) (owner: 10Majavah) [07:36:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [07:36:49] (03CR) 10Volans: [C:03+1] "To be tested but LGTM" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341303 (https://phabricator.wikimedia.org/T437219) (owner: 10Majavah) [07:37:43] (03CR) 10Volans: [C:03+1] "LGTM" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341306 (https://phabricator.wikimedia.org/T437219) (owner: 10Majavah) [07:39:37] (03CR) 10Volans: [C:03+1] "LGTM" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341316 (https://phabricator.wikimedia.org/T437219) (owner: 10Majavah) [07:42:11] (03CR) 10Volans: [C:03+1] "LGTM, question inline" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341304 (https://phabricator.wikimedia.org/T410721) (owner: 10Majavah) [07:44:08] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12321044 (10komla) @dcaro thanks. Yeah, I will just use some of the existing repos for subsequent ones. I converted the `My first PHP tool` tutorial to a Python... [07:48:23] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12321053 (10dcaro) A couple comments, the tool is not running correctly (https://sample-python-ptd-app.toolforge.org gives error). There's python code that's malf... [07:49:52] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12321057 (10komla) Checking .... [07:50:44] (03update) 10dcaro: toolforge_get_versions,components: add logs-cli [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1418 (https://phabricator.wikimedia.org/T432572) [07:51:18] (03merge) 10dcaro: toolforge_get_versions,components: add logs-cli [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1418 (https://phabricator.wikimedia.org/T432572) [07:52:11] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12321062 (10komla) 05Resolved→03Open [08:00:11] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12321075 (10komla) It's working now: https://sample-python-ptd-app.toolforge.org I didn't update the repo url in `toolforge.yaml` file [08:03:32] (03merge) 10saiphani02: T434821 [toolforge-repos/wsindex] - 10https://gitlab.wikimedia.org/toolforge-repos/wsindex/-/merge_requests/1 (owner: 10bodhisattwa) [08:12:58] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12321085 (10komla) Now getting `no healthy upstream` [08:16:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [08:20:27] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12321139 (10dcaro) From the logs: ` tools.sample-python-ptd-app@tools-bastion-15:~$ toolforge jobs logs webservice ... 2026-09-15T08:19:16.399894Z [webservice-55f49... [08:34:19] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12321201 (10komla) Fixed. I've noticed that webservices take a bit more time to build(ie show `status:successful`) than jobs. Is this expected or is it from my end? [08:34:51] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-cli [08:35:23] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-cli [08:35:32] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component logs-cli [08:36:09] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-cli [08:42:42] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12321246 (10aputhin) 05Open→03In progress [08:43:26] (03approved) 10dcaro: d/changelog: bump to 0.0.2 [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/3 [08:43:30] (03merge) 10dcaro: d/changelog: bump to 0.0.2 [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/3 [08:49:21] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12321288 (10komla) Published here: https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Python_tool [08:51:04] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12321306 (10dcaro) >>! In T437899#12321201, @komla wrote: > Fixed. > > I've noticed that webservices take a bit more time to build(ie show `status:successful`) than... [09:03:03] FIRING: InstanceDown: Project tools instance tools-k8s-worker-nfs-16 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [09:09:23] FIRING: ToolforgeKubernetesNodeNotReady: Kubernetes node tools-k8s-worker-nfs-16 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [09:22:43] (03update) 10dcaro: global: add logs-cli support [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/342 (https://phabricator.wikimedia.org/T432572) [10:26:48] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team: Enable use of requestctl for filtering Toolforge and Cloud VPS web traffic - https://phabricator.wikimedia.org/T409759#12321846 (10Joe) >>! In T409759#12295316, @taavi wrote: > @Joe @Raine Hi! AIUI you're the people to talk to about requestctl stuff. We... [10:42:33] 10Tool-Pageviews, 07Regression: Massviews Analysis fails due to 414 Request-URI Too Long - https://phabricator.wikimedia.org/T437947#12321915 (10Fuzzy) Thank you for the quick response and fix. Another improvement might be to pass page_id instead of page_name, if the API supports it. Additionally, consider... [10:43:03] RESOLVED: InstanceDown: Project tools instance tools-k8s-worker-nfs-16 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [10:44:23] RESOLVED: ToolforgeKubernetesNodeNotReady: Kubernetes node tools-k8s-worker-nfs-16 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [10:55:07] 06tools-infrastructure-team, 10Ceph, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): ceph: Fix the CephX key guard so that Puppet can correctly update the key material - https://phabricator.wikimedia.org/T437233#12322025 (10BTullis) [10:57:10] 06tools-infrastructure-team, 10Ceph, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): ceph: Fix the CephX key guard so that Puppet can correctly update the key material - https://phabricator.wikimedia.org/T437233#12322028 (10BTullis) 05Open→03Resolved This is working as expected now. I was able to use... [11:00:35] (03CR) 10Majavah: [C:03+2] etcd: Inline Hiera operations [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341302 (https://phabricator.wikimedia.org/T437219) (owner: 10Majavah) [11:01:08] (03CR) 10Majavah: [C:03+2] etcd: Make Hiera keys configurable [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341303 (https://phabricator.wikimedia.org/T437219) (owner: 10Majavah) [11:01:14] (03CR) 10Majavah: [C:03+2] etcd: Allow specifying network for a new etcd node [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341306 (https://phabricator.wikimedia.org/T437219) (owner: 10Majavah) [11:01:21] (03CR) 10Majavah: [C:03+2] etcd: Allow bootstrapping new clusters [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341316 (https://phabricator.wikimedia.org/T437219) (owner: 10Majavah) [11:06:59] (03Merged) 10jenkins-bot: etcd: Inline Hiera operations [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341302 (https://phabricator.wikimedia.org/T437219) (owner: 10Majavah) [11:07:27] (03CR) 10Majavah: inventory: Add cloudinfra etcd cluster (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341304 (https://phabricator.wikimedia.org/T410721) (owner: 10Majavah) [11:08:48] (03Merged) 10jenkins-bot: etcd: Make Hiera keys configurable [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341303 (https://phabricator.wikimedia.org/T437219) (owner: 10Majavah) [11:09:12] (03Merged) 10jenkins-bot: etcd: Allow specifying network for a new etcd node [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341306 (https://phabricator.wikimedia.org/T437219) (owner: 10Majavah) [11:09:23] (03Merged) 10jenkins-bot: etcd: Allow bootstrapping new clusters [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341316 (https://phabricator.wikimedia.org/T437219) (owner: 10Majavah) [11:28:25] 06cloud-services-team, 10Striker, 06tools-platform-team: Remove unused WIKITECH_USER setting from Striker - https://phabricator.wikimedia.org/T438014 (10Arendpieter) 03NEW [11:29:09] 10Data-Services, 06tools-platform-team, 07User-notice: Update user-facing Wiki Replicas features and documentation for x4 - https://phabricator.wikimedia.org/T437208#12322251 (10taavi) 05Open→03Resolved a:03taavi [11:29:48] (03PS1) 10Arendpieter: config: Remove unused WIKITECH_USER setting [labs/striker] - 10https://gerrit.wikimedia.org/r/1341841 (https://phabricator.wikimedia.org/T438014) [11:38:21] 06tools-infrastructure-team, 10Ceph, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): ceph: Fix the CephX key guard so that Puppet can correctly update the key material - https://phabricator.wikimedia.org/T437233#12322319 (10fgiunchedi) Thank you @BTullis for tackling this, all makes sense to me -- what wo... [12:00:09] (03approved) 10raymond-ndibe: global: add logs-cli support [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/342 (https://phabricator.wikimedia.org/T432572) (owner: 10dcaro) [12:02:31] (03merge) 10dcaro: global: add logs-cli support [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/342 (https://phabricator.wikimedia.org/T432572) [12:08:47] (03CR) 10Volans: [C:03+1] inventory: Add cloudinfra etcd cluster (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341304 (https://phabricator.wikimedia.org/T410721) (owner: 10Majavah) [12:12:40] (03CR) 10Majavah: [C:03+2] inventory: Add cloudinfra etcd cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341304 (https://phabricator.wikimedia.org/T410721) (owner: 10Majavah) [12:15:53] (03Merged) 10jenkins-bot: inventory: Add cloudinfra etcd cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341304 (https://phabricator.wikimedia.org/T410721) (owner: 10Majavah) [12:18:06] (03update) 10raymond-ndibe: cancel: split latest deployment endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/182 (https://phabricator.wikimedia.org/T434499) (owner: 10mahveotm) [12:18:48] (03update) 10raymond-ndibe: cancel: split latest deployment endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/182 (https://phabricator.wikimedia.org/T434499) (owner: 10mahveotm) [12:27:22] (03update) 10raymond-ndibe: cancel: split latest deployment endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/182 (https://phabricator.wikimedia.org/T434499) (owner: 10mahveotm) [12:30:26] (03approved) 10arthurtaylor: Declare the cache value type for pyright [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/14 (owner: 10phedenskog) [12:30:42] (03merge) 10arthurtaylor: Declare the cache value type for pyright [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/14 (owner: 10phedenskog) [12:33:58] (03update) 10raymond-ndibe: cancel: split latest deployment endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/182 (https://phabricator.wikimedia.org/T434499) (owner: 10mahveotm) [12:34:08] (03update) 10arthurtaylor: Load every database split group for database-only jobs [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/13 (https://phabricator.wikimedia.org/T437796) (owner: 10phedenskog) [12:38:08] (03approved) 10arthurtaylor: Load every database split group for database-only jobs [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/13 (https://phabricator.wikimedia.org/T437796) (owner: 10phedenskog) [12:38:13] (03merge) 10arthurtaylor: Load every database split group for database-only jobs [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/13 (https://phabricator.wikimedia.org/T437796) (owner: 10phedenskog) [12:57:50] (03approved) 10raymond-ndibe: cancel: split latest deployment endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/182 (https://phabricator.wikimedia.org/T434499) (owner: 10mahveotm) [12:58:05] (03merge) 10raymond-ndibe: cancel: split latest deployment endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/182 (https://phabricator.wikimedia.org/T434499) (owner: 10mahveotm) [13:01:42] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.216-20260915125825-f33d8805 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1421 (https://phabricator.wikimedia.org/T434499) [13:01:45] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.216-20260915125825-f33d8805 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1421 (https://phabricator.wikimedia.org/T434499) [13:03:49] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.etcd.add_cluster_node [13:10:12] (03merge) 10fnegri: Add ruff and mypy to dev dependencies [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/2 [13:10:15] (03update) 10fnegri: Fetch alerts from Prometheus [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/3 (https://phabricator.wikimedia.org/T432865) [13:12:19] !log taavi@cloudcumin1001 cloudinfra END (FAIL) - Cookbook wmcs.etcd.add_cluster_node (exit_code=99) [13:13:13] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: alerts-api: bump to 0.0.4-20260915131028-71347d7a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1422 [13:13:13] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: alerts-api: bump to 0.0.4-20260915131028-71347d7a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1422 [13:14:50] (03update) 10fnegri: Fetch alerts from Prometheus [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/3 (https://phabricator.wikimedia.org/T432865) [13:16:28] (03update) 10fnegri: alerts-api: configure prometheus hostnames [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1419 (https://phabricator.wikimedia.org/T432865) [13:17:30] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.etcd.remove_cluster_node [13:18:11] !log taavi@cloudcumin1001 cloudinfra END (FAIL) - Cookbook wmcs.etcd.remove_cluster_node (exit_code=99) [13:18:53] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.vps.remove_instance for instance cloudinfra-etcd-2 [13:19:16] !log taavi@cloudcumin1001 cloudinfra END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance cloudinfra-etcd-2 [13:26:59] (03PS1) 10Majavah: etcd: Fix missed Toolforge branding on a cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341881 (https://phabricator.wikimedia.org/T410721) [13:27:02] (03PS1) 10Majavah: etcd: Make sure firewall is always updated [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341882 (https://phabricator.wikimedia.org/T410721) [13:27:17] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.etcd.add_cluster_node [13:33:37] (03PS2) 10Majavah: etcd: Fix missed Toolforge branding on a cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341881 (https://phabricator.wikimedia.org/T437219) [13:33:40] (03PS2) 10Majavah: etcd: Make sure firewall is always updated [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341882 (https://phabricator.wikimedia.org/T437219) [13:36:04] !log taavi@cloudcumin1001 cloudinfra END (FAIL) - Cookbook wmcs.etcd.add_cluster_node (exit_code=99) [13:38:34] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.etcd.add_cluster_node [13:39:00] !log taavi@cloudcumin1001 cloudinfra END (FAIL) - Cookbook wmcs.etcd.add_cluster_node (exit_code=99) [13:40:42] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.etcd.add_cluster_node [13:41:06] !log taavi@cloudcumin1001 cloudinfra END (FAIL) - Cookbook wmcs.etcd.add_cluster_node (exit_code=99) [13:44:17] (03update) 10raymond-ndibe: components-api: bump to 0.0.216-20260915125825-f33d8805 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1421 (https://phabricator.wikimedia.org/T434499) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:44:18] (03approved) 10raymond-ndibe: components-api: bump to 0.0.216-20260915125825-f33d8805 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1421 (https://phabricator.wikimedia.org/T434499) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:44:28] (03merge) 10raymond-ndibe: components-api: bump to 0.0.216-20260915125825-f33d8805 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1421 (https://phabricator.wikimedia.org/T434499) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:58:36] (03CR) 10Volans: [C:03+1] "LGTM" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341881 (https://phabricator.wikimedia.org/T437219) (owner: 10Majavah) [13:59:33] (03CR) 10Volans: [C:03+1] "LGTM" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341882 (https://phabricator.wikimedia.org/T437219) (owner: 10Majavah) [14:21:18] (03open) 10dcaro: global: add deploy events [repos/cloud/toolforge/logs-api] (add-get-events-endpoint) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/42 (https://phabricator.wikimedia.org/T432571) [14:28:50] (03CR) 10Majavah: [C:03+2] etcd: Fix missed Toolforge branding on a cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341881 (https://phabricator.wikimedia.org/T437219) (owner: 10Majavah) [14:28:56] (03CR) 10Majavah: [C:03+2] etcd: Make sure firewall is always updated [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341882 (https://phabricator.wikimedia.org/T437219) (owner: 10Majavah) [14:33:14] (03update) 10fnegri: alerts-api: configure prometheus hostnames [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1419 (https://phabricator.wikimedia.org/T432865) [14:35:13] (03Merged) 10jenkins-bot: etcd: Fix missed Toolforge branding on a cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341881 (https://phabricator.wikimedia.org/T437219) (owner: 10Majavah) [14:35:15] (03Merged) 10jenkins-bot: etcd: Make sure firewall is always updated [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341882 (https://phabricator.wikimedia.org/T437219) (owner: 10Majavah) [14:42:47] (03update) 10tlepage: refactor: Generalize Loki push format [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/41 (https://phabricator.wikimedia.org/T432566) [14:52:36] 10Toolforge, 06tools-platform-team: [components-api] split cancel endpoint to avoid duplicated operation id - https://phabricator.wikimedia.org/T434499#12323387 (10Raymond_Ndibe) 05In progress→03Resolved [15:20:40] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.etcd.add_cluster_node [15:21:04] !log taavi@cloudcumin1001 cloudinfra END (FAIL) - Cookbook wmcs.etcd.add_cluster_node (exit_code=99) [15:22:43] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.etcd.add_cluster_node [15:23:08] !log taavi@cloudcumin1001 cloudinfra END (FAIL) - Cookbook wmcs.etcd.add_cluster_node (exit_code=99) [15:35:17] 10Cloud-VPS, 06tools-infrastructure-team: Establish an automated cinder volume restore procedure and testing - https://phabricator.wikimedia.org/T438051 (10fgiunchedi) 03NEW [15:40:42] (03update) 10dcaro: support --webservice option [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/143 (https://phabricator.wikimedia.org/T428898) (owner: 10raymond-ndibe) [15:42:33] (03open) 10fnegri: Keep polling when a prometheus host is unreachable [repos/cloud/toolforge/alerts-api] (fran-T432865-505040f5) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/4 (https://phabricator.wikimedia.org/T432865) [15:43:45] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12323751 (10komla) Yes, jobs with publish like this one. Perhaps, I'm just perceiving it even though it may not be the case. I'll test a bit more. [15:46:52] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12323768 (10komla) 05In progress→03Resolved [15:49:37] (03PS1) 10Majavah: vps: create_instance: Add availability zone flag [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341932 [15:49:37] (03PS1) 10Majavah: etcd: Allow setting --availability-zone [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341933 [15:50:31] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.etcd.add_cluster_node [15:50:56] !log taavi@cloudcumin1001 cloudinfra END (FAIL) - Cookbook wmcs.etcd.add_cluster_node (exit_code=99) [15:54:31] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.etcd.add_cluster_node [15:57:57] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12323917 (10dcaro) @komla the code snippets here https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Python_tool are still malformed, did you move the tut... [16:00:27] 10Cloud-VPS (Debian Bullseye Deprecation), 10Continuous-Integration-Infrastructure: Re-build integration-cumin.integration.eqiad1.wikimedia.cloud on something newer than bullseye - https://phabricator.wikimedia.org/T433592#12323927 (10komla) Hello @Jdforrester-WMF and @hashar any update on this? [16:01:47] (03CR) 10Volans: [C:03+1] "LGTM" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341933 (owner: 10Majavah) [16:03:20] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [logs-cli,toolforge-cli] Add subcommand toolforge logs to query and retrieve the relevant logs, will have to design a minimal interface to filter by event/subsystem/time/etc. - https://phabricator.wikimedia.org/T432572#12323936 (10dcaro) ... [16:03:40] !log taavi@cloudcumin1001 cloudinfra END (PASS) - Cookbook wmcs.etcd.add_cluster_node (exit_code=0) [16:06:15] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12323960 (10komla) 05Resolved→03Open [16:07:51] (03PS2) 10Majavah: etcd: Allow setting --availability-zone [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341933 [16:08:16] (03CR) 10Majavah: etcd: Allow setting --availability-zone (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341933 (owner: 10Majavah) [16:09:11] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team: Make Toolforge etcd cookbooks more generic so they are usable for the new cloudinfra cluster - https://phabricator.wikimedia.org/T437219#12323976 (10taavi) 05Open→03Resolved [16:09:41] (03update) 10fnegri: Keep polling when a prometheus host is unreachable [repos/cloud/toolforge/alerts-api] (fran-T432865-505040f5) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/4 (https://phabricator.wikimedia.org/T432865) [16:10:45] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12323998 (10komla) Do you mean they are like this {F103044321} Instead of this: {F103044488} [16:12:05] (03CR) 10Volans: [C:03+1] "Thanks" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341933 (owner: 10Majavah) [16:13:19] (03update) 10fnegri: Keep polling when a prometheus host is unreachable [repos/cloud/toolforge/alerts-api] (fran-T432865-505040f5) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/4 (https://phabricator.wikimedia.org/T432865) [16:14:06] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12324008 (10komla) I'm using the I will use the code sample template then. [16:14:19] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12324010 (10dcaro) This is how that link show for me: {F103045289} [16:15:24] (03CR) 10Volans: [C:03+1] "LGTM" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341932 (owner: 10Majavah) [16:15:55] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12324041 (10komla) I see. Interesting! Let me use the code sample template instead of the `syntaxhighlight` tag. [16:18:28] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12324071 (10dcaro) You just have to do the same you just did with the other code: https://wikitech.wikimedia.org/w/index.php?title=Help%3AToolforge%2FMy_first_Python_... [16:18:45] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12324072 (10komla) I've changed that specific code block. Please confirm before I proceed with the other. [16:20:11] (03CR) 10Majavah: [C:03+2] vps: create_instance: Add availability zone flag [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341932 (owner: 10Majavah) [16:20:16] (03CR) 10Majavah: [C:03+2] etcd: Allow setting --availability-zone [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341933 (owner: 10Majavah) [16:20:21] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12324086 (10dcaro) >>! In T437899#12324072, @komla wrote: > I've changed that specific code block. > > Please confirm before I proceed with the others. Nope, you ha... [16:20:56] 10Cloud-VPS (Debian Bullseye Deprecation), 10Continuous-Integration-Infrastructure: Re-build integration-cumin.integration.eqiad1.wikimedia.cloud on something newer than bullseye - https://phabricator.wikimedia.org/T433592#12324089 (10Jdforrester-WMF) Sorry, no, I've not had time to work on this. @hashar, can... [16:21:12] (03PS7) 10Arendpieter: Use IDP for authentication [labs/striker] - 10https://gerrit.wikimedia.org/r/1250537 (https://phabricator.wikimedia.org/T359554) [16:22:42] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12324095 (10komla) That's weird. That previous change are identation changes. This is what I'm seeing now: {F103046987} [16:23:38] (03Merged) 10jenkins-bot: vps: create_instance: Add availability zone flag [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341932 (owner: 10Majavah) [16:23:59] (03Merged) 10jenkins-bot: etcd: Allow setting --availability-zone [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341933 (owner: 10Majavah) [16:24:24] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12324099 (10komla) Does the code block still show as black background? [16:24:54] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12324103 (10komla) I've added the indentation to the 'html' in the `return` statement [16:37:23] 10PAWS, 10Toolforge, 06tools-platform-team: Dumps NFS mount down - https://phabricator.wikimedia.org/T438062 (10Pppery) 03NEW [16:46:17] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12324219 (10dcaro) >>! In T437899#12324099, @komla wrote: > Does the code block still show as black background? The background is because I chose dark theme, that's... [16:46:53] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12324229 (10dcaro) You can test it out if it works yourself by copy-pasting the code when creating the tool. [16:50:46] (03CR) 10Andrew Bogott: "one helpstring request below but I'm fine with this being merged already" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341932 (owner: 10Majavah) [16:55:32] 10Toolforge, 06tools-platform-team, 06DBA: Access denied for user 's54673'@'%' to database 'ruwiki_p' - https://phabricator.wikimedia.org/T438068 (10MBH) 03NEW [16:56:27] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12324284 (10komla) Yeah, it does. No syntax errors. ` komla@EDOR-MACHINE:~/src/wmf/___tests$ python3 -m py_compile app.py komla@EDOR-MACHINE:~/src/wmf/___tests$ ` [17:01:04] 10Toolforge, 06tools-platform-team, 06DBA: Access denied for user 's54673'@'%' to database 'ruwiki_p' - https://phabricator.wikimedia.org/T438068#12324324 (10taavi) The credentials in the tool home work correctly: `lines=5 tools.wikisaurusbot@tools-bastion-14:~$ sql ruwiki Reading table information for compl... [17:01:34] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12324326 (10dcaro) https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Python_tool still shows wrong for me: {F103053797} I don't have much time right now, s... [17:03:50] (03open) 10dcaro: auth: generalize the writer authorization [repos/cloud/toolforge/logs-api] (add-get-events-endpoint) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/43 (https://phabricator.wikimedia.org/T432571) [17:04:52] 10Toolforge, 06tools-platform-team, 06DBA: Access denied for user 's54673'@'%' to database 'ruwiki_p' - https://phabricator.wikimedia.org/T438068#12324340 (10MBH) ` const BASE_URL = 'https://ru.wikipedia.org'; const configPath = path.resolve( __dirname, process.env.DB_CNF_PATH || '../../../../replica.my.c... [17:05:06] 10Toolforge, 06tools-platform-team: Access denied for user 's54673'@'%' to database 'ruwiki_p' - https://phabricator.wikimedia.org/T438068#12324342 (10Ladsgroup) DBA tag is for production databases, I stay subscribed in case it's needed. [17:10:05] (03PS1) 10Majavah: etcd: Support managing a SRV DNS record for etcd cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341953 [17:11:25] 10PAWS, 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: Dumps NFS mount down - https://phabricator.wikimedia.org/T438062#12324390 (10BLiviero-WMF) [17:12:57] (03approved) 10raymond-ndibe: refactor: Generalize Loki push format [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/41 (https://phabricator.wikimedia.org/T432566) (owner: 10tlepage) [17:13:08] (03CR) 10CI reject: [V:04-1] etcd: Support managing a SRV DNS record for etcd cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341953 (owner: 10Majavah) [17:22:02] (03PS2) 10Majavah: etcd: Support managing a SRV DNS record for etcd cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341953 [17:22:14] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.etcd.remove_cluster_node [17:22:57] !log taavi@cloudcumin1001 cloudinfra END (FAIL) - Cookbook wmcs.etcd.remove_cluster_node (exit_code=99) [17:24:21] (03PS3) 10Majavah: etcd: Support managing a SRV DNS record for etcd cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341953 [17:24:27] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.etcd.remove_cluster_node [17:24:34] 10Toolforge, 06tools-platform-team: Access denied for user 's54673'@'%' to database 'ruwiki_p' - https://phabricator.wikimedia.org/T438068#12324437 (10MBH) 05Open→03Resolved a:03MBH Looks like it works if replace 'ukwiki.analytics.db.svc.**eqiad.wmflabs**' to 'ukwiki.analytics.db.svc.**wikimedia.clou... [17:25:29] !log taavi@cloudcumin1001 cloudinfra END (PASS) - Cookbook wmcs.etcd.remove_cluster_node (exit_code=0) [17:26:04] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.etcd.add_cluster_node [17:26:36] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12324450 (10komla) Thanks @dcaro All this while, I was focused on the longer `app.py` code block not this shorter version. It's formatted now. [17:27:31] 10Toolforge, 06tools-platform-team: Access denied for user 's54673'@'%' to database 'ruwiki_p' - https://phabricator.wikimedia.org/T438068#12324456 (10taavi) 05Resolved→03Invalid Marking as invalid since there was no infrastructure issue. For generic support see https://wikitech.wikimedia.org/wiki/Help... [17:34:21] (03update) 10tlepage: events: Add the GET /{tool}/logs endpoint [repos/cloud/toolforge/logs-api] (loki-refactor-for-get-events) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) [17:35:12] !log taavi@cloudcumin1001 cloudinfra END (PASS) - Cookbook wmcs.etcd.add_cluster_node (exit_code=0) [17:37:03] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance cloudinfra-etcd-3 in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [17:39:38] 10Tool-wsindex, 06MediaWiki-Media-Platform-Team, 10Wikipedia-iOS-App, 06WikipediaApp, and 2 others: Update apps to support for the move from upload.wikimedia.org to thumb.wikimedia.org - https://phabricator.wikimedia.org/T434821#12324533 (10HNordeenWMF) [17:40:41] (03update) 10tlepage: events: Add the GET /{tool}/logs endpoint [repos/cloud/toolforge/logs-api] (loki-refactor-for-get-events) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) [17:41:06] (03update) 10tlepage: events: Add the GET /{tool}/logs endpoint [repos/cloud/toolforge/logs-api] (loki-refactor-for-get-events) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) [17:43:37] (03merge) 10tlepage: refactor: Generalize Loki push format [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/41 (https://phabricator.wikimedia.org/T432566) [17:43:39] (03update) 10tlepage: refactor: Parse Loki result streams into logs [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T432566) [17:46:34] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.49-20260915174347-1aba6be1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1423 [17:46:35] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.49-20260915174347-1aba6be1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1423 [17:47:29] !log tlepage@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-api [17:57:03] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance cloudinfra-etcd-3 in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [17:58:15] !log tlepage@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [17:59:01] !log tlepage@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component logs-api [18:01:52] 10Cloud-VPS (Debian Bullseye Deprecation), 10a Wikimedia CLI: Migrate mwcli away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435778#12324595 (10Addshore) I'll be looking at it today :) [18:02:39] 10Cloud-VPS (Debian Bullseye Deprecation), 10a Wikimedia CLI: Migrate mwcli away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435778#12324596 (10Addshore) > this project will be shut down on Tuesday, September 1st. And if I don't manage today, please just go ahead and shut down... [18:02:55] 10Cloud-VPS (Debian Bullseye Deprecation), 10a Wikimedia CLI: Migrate mwcli away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435778#12324597 (10Addshore) a:03Addshore [18:09:13] !log tlepage@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [18:10:40] (03merge) 10tlepage: logs-api: bump to 0.0.49-20260915174347-1aba6be1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1423 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [18:12:35] 10PAWS, 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: Dumps NFS mount down - https://phabricator.wikimedia.org/T438062#12324670 (10fgiunchedi) Not sure yet of the cause but flipping all nfs traffic to clouddumps1002 looks like it has restored service [18:17:57] 10PAWS, 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: Dumps NFS mount down - https://phabricator.wikimedia.org/T438062#12324689 (10fgiunchedi) Things seem to be stable with nfs-kernel-server restarted on clouddump1002 and all traffic flipped to it. Proper investigation and followups pend... [19:27:58] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06Traffic: Replace cache proxies: deployment-cache-(text|upload)08 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436468#12324916 (10Southparkfan) Hi @BCornwall, thank you for your help! I have added the servers to `cache_... [19:33:01] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace Acme-chief hosts: deployment-acme-chief0[56] (Bullseye deprecation) - https://phabricator.wikimedia.org/T436472#12324940 (10Southparkfan) Hi @BCornwall you have created `deployment-acme-chief07` and `deployment-acme-chief08`, and... [19:37:30] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06Traffic, 13Patch-For-Review: Replace cache proxies: deployment-cache-(text|upload)08 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436468#12324962 (10BCornwall) Thanks, @Southparkfan ! I went ahead and moved the float... [20:29:42] 10Toolforge: Access denied for user 's54673'@'%' to database 'ruwiki_p' - https://phabricator.wikimedia.org/T438068#12325125 (10aputhin) [20:29:54] 10PAWS, 10Toolforge, 06tools-infrastructure-team: Dumps NFS mount down - https://phabricator.wikimedia.org/T438062#12325126 (10aputhin) [20:51:31] 10wikitech.wikimedia.org, 10Parsoid, 10Parsoid-Read-Views, 06Content-Platform-Team (Work In Progress), 13Patch-For-Review: Parsoid Read Views: Deploy to wikitech - https://phabricator.wikimedia.org/T437916#12325195 (10ABreault-WMF) [20:54:11] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06Traffic: Replace cache proxies: deployment-cache-(text|upload)08 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436468#12325202 (10BCornwall) Yay, thanks for the help deploying it. Regarding tofu integration, I'm not see... [21:02:17] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace Acme-chief hosts: deployment-acme-chief0[56] (Bullseye deprecation) - https://phabricator.wikimedia.org/T436472#12325213 (10BCornwall) 05Open→03Resolved a:03BCornwall [21:04:02] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12325217 (10aputhin) p:05Triage→03Low [21:37:59] 10Cloud-VPS (Debian Bullseye Deprecation), 10a Wikimedia CLI: Migrate mwcli away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435778#12325307 (10Addshore) 05Open→03Resolved Managed to setup `gitlab-runner-addshore-1018` on the project using Trixie and all went according... [21:38:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:33:17] FIRING: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [23:03:17] RESOLVED: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [23:05:02] FIRING: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [23:09:47] RESOLVED: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [23:33:17] FIRING: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [23:48:41] RESOLVED: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks