[00:16:28] (InstanceDown) firing: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:21:28] (InstanceDown) resolved: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [01:36:39] (ProbeDown) firing: (3) Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:41:39] (ProbeDown) resolved: (4) Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:24:35] (03PS1) 10Krinkle: Rewrite FileProtectionSyncBot in PHP 8 with zero dependencies [labs/tools/fileprotectionsync] - 10https://gerrit.wikimedia.org/r/1015661 [02:24:44] (03PS2) 10Krinkle: Rewrite FileProtectionSyncBot in PHP 8 with zero dependencies [labs/tools/fileprotectionsync] - 10https://gerrit.wikimedia.org/r/1015661 [02:25:30] (03CR) 10CI reject: [V:04-1] Rewrite FileProtectionSyncBot in PHP 8 with zero dependencies [labs/tools/fileprotectionsync] - 10https://gerrit.wikimedia.org/r/1015661 (owner: 10Krinkle) [02:26:58] (03PS3) 10Krinkle: Rewrite FileProtectionSyncBot in PHP 8 with zero dependencies [labs/tools/fileprotectionsync] - 10https://gerrit.wikimedia.org/r/1015661 [02:27:50] (03CR) 10CI reject: [V:04-1] Rewrite FileProtectionSyncBot in PHP 8 with zero dependencies [labs/tools/fileprotectionsync] - 10https://gerrit.wikimedia.org/r/1015661 (owner: 10Krinkle) [02:39:01] (03PS4) 10Krinkle: Rewrite FileProtectionSyncBot in PHP 8 with zero dependencies [labs/tools/fileprotectionsync] - 10https://gerrit.wikimedia.org/r/1015661 [02:40:14] (03CR) 10CI reject: [V:04-1] Rewrite FileProtectionSyncBot in PHP 8 with zero dependencies [labs/tools/fileprotectionsync] - 10https://gerrit.wikimedia.org/r/1015661 (owner: 10Krinkle) [02:40:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:41:25] (03CR) 10Krinkle: "recheck" [labs/tools/fileprotectionsync] - 10https://gerrit.wikimedia.org/r/1015661 (owner: 10Krinkle) [02:43:25] (03PS5) 10Krinkle: Rewrite FileProtectionSyncBot in PHP 8 with zero dependencies [labs/tools/fileprotectionsync] - 10https://gerrit.wikimedia.org/r/1015661 [02:45:09] (03CR) 10Krinkle: [C:03+2] Rewrite FileProtectionSyncBot in PHP 8 with zero dependencies [labs/tools/fileprotectionsync] - 10https://gerrit.wikimedia.org/r/1015661 (owner: 10Krinkle) [02:45:41] (CloudVPSDesignateLeaks) firing: (4) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:45:46] (03Merged) 10jenkins-bot: Rewrite FileProtectionSyncBot in PHP 8 with zero dependencies [labs/tools/fileprotectionsync] - 10https://gerrit.wikimedia.org/r/1015661 (owner: 10Krinkle) [02:46:07] (03PS1) 10Krinkle: Fix missing yaml_parse() [labs/tools/fileprotectionsync] - 10https://gerrit.wikimedia.org/r/1015686 [02:50:17] (03CR) 10Krinkle: [C:03+2] Fix missing yaml_parse() [labs/tools/fileprotectionsync] - 10https://gerrit.wikimedia.org/r/1015686 (owner: 10Krinkle) [02:50:23] (03PS2) 10Krinkle: Fix missing yaml_parse() [labs/tools/fileprotectionsync] - 10https://gerrit.wikimedia.org/r/1015686 [02:50:27] (03CR) 10Krinkle: [C:03+2] Fix missing yaml_parse() [labs/tools/fileprotectionsync] - 10https://gerrit.wikimedia.org/r/1015686 (owner: 10Krinkle) [02:51:27] (03Merged) 10jenkins-bot: Fix missing yaml_parse() [labs/tools/fileprotectionsync] - 10https://gerrit.wikimedia.org/r/1015686 (owner: 10Krinkle) [03:00:16] (OpenstackAPIResponse) firing: (2) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [03:15:41] (CloudVPSDesignateLeaks) firing: (5) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:20:41] (CloudVPSDesignateLeaks) firing: (5) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:21:36] (03PS1) 10Krinkle: write_config: index operations/docker-images/toollabs-images [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1015689 [04:22:48] (03CR) 10CI reject: [V:04-1] write_config: index operations/docker-images/toollabs-images [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1015689 (owner: 10Krinkle) [04:25:41] (CloudVPSDesignateLeaks) resolved: (5) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:25:48] 10Toolforge: Install php-yaml in Toolforge images - https://phabricator.wikimedia.org/T361457 (10Krinkle) 03NEW [04:26:06] (03PS2) 10Krinkle: write_config: index operations/docker-images/toollabs-images [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1015689 (https://phabricator.wikimedia.org/T361457) [04:26:53] (03PS3) 10Krinkle: write_config: index operations/docker-images/toollabs-images [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1015689 (https://phabricator.wikimedia.org/T361457) [04:38:15] (03CR) 10Krinkle: [C:03+2] write_config: index operations/docker-images/toollabs-images [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1015689 (https://phabricator.wikimedia.org/T361457) (owner: 10Krinkle) [04:39:39] (03Merged) 10jenkins-bot: write_config: index operations/docker-images/toollabs-images [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1015689 (https://phabricator.wikimedia.org/T361457) (owner: 10Krinkle) [04:41:32] 10Toolforge (Software install/update): Install php-yaml in Toolforge images - https://phabricator.wikimedia.org/T361457#9675454 (10JJMC89) [04:44:12] (03PS1) 10Krinkle: php74-sssd,php82-sssd: add php-yaml [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1015690 (https://phabricator.wikimedia.org/T361457) [04:56:10] (03PS1) 10Krinkle: Temporarily freeze logo protection [labs/tools/fileprotectionsync] - 10https://gerrit.wikimedia.org/r/1015691 (https://phabricator.wikimedia.org/T361457) [04:56:23] (03CR) 10Krinkle: [C:03+2] Temporarily freeze logo protection [labs/tools/fileprotectionsync] - 10https://gerrit.wikimedia.org/r/1015691 (https://phabricator.wikimedia.org/T361457) (owner: 10Krinkle) [04:56:56] (03Merged) 10jenkins-bot: Temporarily freeze logo protection [labs/tools/fileprotectionsync] - 10https://gerrit.wikimedia.org/r/1015691 (https://phabricator.wikimedia.org/T361457) (owner: 10Krinkle) [05:00:09] 06Toolforge-standards-committee: Adoption request for Yapperbot - https://phabricator.wikimedia.org/T361426#9675464 (10Legoktm) >>! In T361426#9674806, @bd808 wrote: > The bot seems to be actively editing (https://en.wikipedia.org/wiki/Special:Contributions/Yapperbot) which means the tool is not obviously meetin... [05:10:45] (ProbeDown) firing: Service tools-legacy-redirector-2:443 has failed probes (http_toolserver_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:15:45] (ProbeDown) resolved: Service tools-legacy-redirector-2:443 has failed probes (http_toolserver_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:43:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:43:45] 10Data-Services, 06DBA: 14Experiment with InnoDB buffer pool size on clouddb1019.eqiad.wmnet - 14https://phabricator.wikimedia.org/T346464#9675485 (10Marostegui) 05Open→03Declined 14I am going to decline this for now as there seem not to be a clear path for this as the ownership isn't clear. Please re... [05:58:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:08:56] (SystemdUnitDown) firing: (2) The service unit wikitech_run_jobs.service is in failed status on host cloudweb1003. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [06:23:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:23:56] (SystemdUnitDown) resolved: (2) The service unit wikitech_run_jobs.service is in failed status on host cloudweb1003. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [06:45:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:00:16] (OpenstackAPIResponse) firing: (2) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [07:15:01] (OpenstackAPIResponse) firing: (2) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [07:25:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:15:40] 10Quarry, 10Toolforge, 10ChangeProp, 06collaboration-services, and 9 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9675600 (10Aklapper) As API Gateway is nowadays owned by #ServiceOps, adding the #serviceops project tag to open AP... [08:17:39] (ProbeDown) firing: (3) Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:22:39] (ProbeDown) resolved: (4) Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:05:01] (OpenstackAPIResponse) resolved: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [09:40:41] (CloudVPSDesignateLeaks) firing: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:45:41] (CloudVPSDesignateLeaks) firing: (5) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:50:41] (CloudVPSDesignateLeaks) firing: (5) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:55:41] (CloudVPSDesignateLeaks) resolved: (5) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:43:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:53:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:58:41] (CloudVPSDesignateLeaks) resolved: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:15:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:15:45] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review: [jobs-cli] Allow exporting jobs list in YAML format - https://phabricator.wikimedia.org/T320575#9675833 (10aborrero) patch should be ready for final review & merge. [11:17:26] (03CR) 10Jforrester: [C:03+2] "Neat." [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1015639 (owner: 10Legoktm) [11:17:34] (03Merged) 10jenkins-bot: Monitor APCu releases for Docker-Hub-MediaWiki [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1015639 (owner: 10Legoktm) [11:25:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:32:41] 06cloud-services-team: Supporting AI, LLM, and data models on WMCS - https://phabricator.wikimedia.org/T336905#9675857 (10aborrero) [12:05:42] 10Quarry: Deploy magnum cluster for quarry - https://phabricator.wikimedia.org/T349032#9675910 (10rook) [12:29:44] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/1015966 (owner: 10L10n-bot) [12:47:44] vivian-rook closed https://github.com/toolforge/quarry/pull/31 [12:47:56] 10Quarry: Deploy magnum cluster for quarry - https://phabricator.wikimedia.org/T349032#9676034 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/quarry/pull/31 [12:48:01] 10Quarry: Deploy magnum cluster for quarry - https://phabricator.wikimedia.org/T349032#9676035 (10rook) Quarry is now on kubernetes. [12:49:02] 10Quarry: 14Deploy magnum cluster for quarry - 14https://phabricator.wikimedia.org/T349032#9676038 (10rook) 05Open→03Resolved [12:57:53] 10Quarry, 10Toolforge, 10ChangeProp, 06collaboration-services, and 9 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9676049 (10akosiaris) LWN has an article titled "The race to replace Redis". I am not going to link directly as it... [13:05:41] 10Quarry, 10Toolforge, 10ChangeProp, 06collaboration-services, and 9 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9676057 (10aborrero) [13:25:54] 10Quarry: Shutdown quarry VMs - https://phabricator.wikimedia.org/T361470 (10rook) 03NEW [13:25:56] 10Quarry: Shutdown quarry VMs - https://phabricator.wikimedia.org/T361470#9676085 (10rook) [13:25:57] 10Quarry: 14Deploy magnum cluster for quarry - 14https://phabricator.wikimedia.org/T349032#9676084 (10rook) [13:34:45] 10Quarry: Quarry login fails due to redirect to plaintext HTTP URL - https://phabricator.wikimedia.org/T361471 (10taavi) 03NEW [13:39:28] 10Quarry: Quarry login fails due to redirect to plaintext HTTP URL - https://phabricator.wikimedia.org/T361471#9676116 (10rook) I'm not able to reproduce this in firefox 124.0.1. Do I need to do anything additional to reproduce? [13:48:14] 10Quarry: Quarry login fails due to redirect to plaintext HTTP URL - https://phabricator.wikimedia.org/T361471#9676118 (10taavi) Can you try with Firefox "HTTPS-Only Mode" (in the "Privacy & Security" about:preferences tab) enabled? [13:52:23] 10Quarry: Quarry login fails due to redirect to plaintext HTTP URL - https://phabricator.wikimedia.org/T361471#9676122 (10rook) Strangely still seems to be letting log in with HTTPS-Only Mode enabled, restarted firefox just in case, still can log in [13:53:44] 10Quarry: Quarry login fails due to redirect to plaintext HTTP URL - https://phabricator.wikimedia.org/T361471#9676129 (10rook) Regardless, I'm unsure of where that location header is populated from. There is no reason for it to be http. [14:13:11] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [14:17:04] 10Quarry: Quarry login fails due to redirect to plaintext HTTP URL - https://phabricator.wikimedia.org/T361471#9676169 (10taavi) My gueus is that some proxy in the middle is not relaying the `x-forwarded-proto` header to the backend correctly. [14:19:45] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) [14:25:28] (InstanceDown) firing: Project toolsbeta instance toolsbeta-test-k8s-etcd-23 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:25:36] 10Quarry: Quarry login fails due to redirect to plaintext HTTP URL - https://phabricator.wikimedia.org/T361471#9676202 (10rook) I'm not sure how the backend saw it as https before. As the web proxy terminates tls and delivered to http://172.16.5.58:80. Though I suppose it must have delivered x-forwarded-proto al... [14:30:28] (InstanceDown) resolved: Project toolsbeta instance toolsbeta-test-k8s-etcd-23 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:30:36] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T349207) [14:30:39] T349207: [infra] Upgrade Toolforge K8s etcd nodes to Bullseye - https://phabricator.wikimedia.org/T349207 [14:40:46] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) [14:42:37] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [14:49:58] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) [14:56:28] (InstanceDown) firing: Project toolsbeta instance toolsbeta-test-k8s-etcd-23 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:59:59] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T349207) [15:00:03] T349207: [infra] Upgrade Toolforge K8s etcd nodes to Bullseye - https://phabricator.wikimedia.org/T349207 [15:01:28] (InstanceDown) resolved: Project toolsbeta instance toolsbeta-test-k8s-etcd-23 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [15:11:20] !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) [15:13:37] 10Data-Services, 06DBA: 14Experiment with InnoDB buffer pool size on clouddb1019.eqiad.wmnet - 14https://phabricator.wikimedia.org/T346464#9676286 (10dr0ptp4kt) 14Makes sense to shelve for now @Marostegui. [15:18:53] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [15:25:02] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [15:30:31] (03CR) 10BryanDavis: [C:03+1] "If we could only trust the jerk who maintains this PHP extension..." [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1015690 (https://phabricator.wikimedia.org/T361457) (owner: 10Krinkle) [15:36:11] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: requests to increase quotas deployment-prep - https://phabricator.wikimedia.org/T361477 (10thcipriani) 03NEW [15:37:06] 10Cloud-VPS (Quota-requests), 10Beta-Cluster-Infrastructure: requests to increase quotas deployment-prep - https://phabricator.wikimedia.org/T361477#9676354 (10JJMC89) [15:37:46] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Migrate deployment-prep away from Debian Buster to Bullseye/Bookworm - https://phabricator.wikimedia.org/T327742#9676352 (10thcipriani) >>! In T327742#9673515, @thcipriani wrote: >>>! In T327742#9673367, @thcipriani wrote: >> * Quota: depl... [15:48:05] !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [15:49:51] !log andrew@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [16:15:28] (PuppetAgentFailure) firing: Puppet agent failure detected on instance toolsbeta-test-k8s-etcd-23 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [16:18:22] 06cloud-services-team, 10Toolforge: Find a modern hostname for tools-static.wmflabs.org - https://phabricator.wikimedia.org/T361435#9676469 (10Bawolff) >>! In T361435#9675012, @bd808 wrote: >>>! In T361435#9674842, @Bawolff wrote: >> If its being moved anyways, would be kind of cool if it was sub-domain based.... [16:26:29] (PuppetAgentNoResources) firing: No Puppet resources found on instance toolsbeta-test-localdisk on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [16:30:28] (PuppetAgentFailure) firing: (2) Puppet agent failure detected on instance toolsbeta-test-k8s-etcd-22 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [16:35:28] (PuppetAgentFailure) firing: (3) Puppet agent failure detected on instance toolsbeta-test-k8s-etcd-20 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [16:40:27] 10Cloud-VPS (Debian Buster Deprecation), 10LibUp: Upgrade LibUp worker to Bookworm - https://phabricator.wikimedia.org/T361488 (10taavi) 03NEW [16:43:49] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:45:28] (PuppetAgentFailure) firing: (3) Puppet agent failure detected on instance toolsbeta-test-k8s-etcd-20 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [16:50:28] (PuppetAgentFailure) firing: (3) Puppet agent failure detected on instance toolsbeta-test-k8s-etcd-20 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [17:08:27] 10Cloud-VPS (Debian Buster Deprecation), 10LibUp: Upgrade LibUp worker to Bookworm - https://phabricator.wikimedia.org/T361488#9676738 (10taavi) [17:08:42] 10Cloud-VPS (Debian Buster Deprecation), 10LibUp: Upgrade LibUp worker to Bookworm - https://phabricator.wikimedia.org/T361488#9676733 (10taavi) [17:11:55] 10Cloud-VPS (Debian Buster Deprecation), 10LibUp, 13Patch-For-Review: Upgrade LibUp worker to Bookworm - https://phabricator.wikimedia.org/T361488#9676786 (10CodeReviewBot) taavi merged https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/25 Update dependency lock file for Bookworm [17:11:56] 10Cloud-VPS (Debian Buster Deprecation), 10LibUp, 13Patch-For-Review: Upgrade LibUp worker to Bookworm - https://phabricator.wikimedia.org/T361488#9676766 (10CodeReviewBot) taavi opened https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/25 Update dependency lock file for Bookworm [17:13:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:15:22] 10Cloud-VPS (Debian Buster Deprecation), 10LibUp, 13Patch-For-Review: Upgrade LibUp worker to Bookworm - https://phabricator.wikimedia.org/T361488#9676789 (10taavi) `libup-runner08` is now up based the Puppet code in the `production` branch. Now I'm waiting for the queues to clear on `upgrader-06` before sta... [17:18:41] (CloudVPSDesignateLeaks) firing: (5) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:12:43] 10PAWS: Upgrade Rstudio - https://phabricator.wikimedia.org/T361506 (10rook) 03NEW [18:14:14] 10Quarry, 10Toolforge, 10ChangeProp, 06collaboration-services, and 9 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9677129 (10Tgr) [18:14:20] vivian-rook opened https://github.com/toolforge/paws/pull/395 [18:14:25] 10PAWS: Upgrade Rstudio - https://phabricator.wikimedia.org/T361506#9677141 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/395 [18:30:53] 10Quarry, 10Toolforge, 10ChangeProp, 06collaboration-services, and 9 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9677197 (10Tgr) [18:37:41] 10PAWS: 14Upgrade Rstudio - 14https://phabricator.wikimedia.org/T361506#9677216 (10github-toolforge-bot) 14vivian-rook closed https://github.com/toolforge/paws/pull/395 [18:37:42] 10PAWS: 14Upgrade Rstudio - 14https://phabricator.wikimedia.org/T361506#9677217 (10rook) 05Open→03Resolved [18:37:49] vivian-rook closed https://github.com/toolforge/paws/pull/395 [18:59:52] 10VPS-project-Wikistats: Add bewwiki to wikistats - https://phabricator.wikimedia.org/T360314#9677417 (10Dzahn) 05Open→03Stalled [19:00:05] 10VPS-project-Wikistats: Add kuswiki to wikistats - https://phabricator.wikimedia.org/T360307#9677421 (10Dzahn) 05Open→03Stalled [19:42:28] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Beta-Cluster-Infrastructure: requests to increase quotas deployment-prep - https://phabricator.wikimedia.org/T361477#9677547 (10bd808) +1 [19:42:30] 10Cloud-VPS (Debian Buster Deprecation), 10LibUp: 14Upgrade LibUp worker to Bookworm - 14https://phabricator.wikimedia.org/T361488#9677550 (10taavi) 05Open→03Resolved [19:46:50] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Beta-Cluster-Infrastructure: 14requests to increase quotas deployment-prep - 14https://phabricator.wikimedia.org/T361477#9677556 (10rook) 14C'est fait [19:46:52] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Beta-Cluster-Infrastructure: 14requests to increase quotas deployment-prep - 14https://phabricator.wikimedia.org/T361477#9677557 (10rook) 05Open→03Resolved a:03rook [20:12:28] (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance enc-1 in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [20:13:57] 10cloud-services-team (FY2023/2024-Q3-Q4), 06Infrastructure-Foundations, 10Spicerack, 10SRE-tools, 10Data-Platform-SRE (2024.03.25 - 2024.04.14): create and deploy new Elastic Curator deb package - https://phabricator.wikimedia.org/T361105#9677642 (10bking) [20:14:57] 10Cloud-VPS: Request to add catalyst-qte.wmcloud.org webproxy subdomain for the catalyst-qte CloudVPS project - https://phabricator.wikimedia.org/T361517 (10EBomani) 03NEW [20:26:38] 10Wikibugs: Replace Redis queue with custom http solution - https://phabricator.wikimedia.org/T361518 (10bd808) 03NEW [20:27:05] 10Wikibugs: Replace Redis queue with custom http solution - https://phabricator.wikimedia.org/T361518#9677689 (10bd808) 05Open→03In progress p:05Triage→03Medium a:03bd808 [20:32:28] (PuppetAgentStaleLastRun) firing: (3) Last Puppet run was over 24 hours ago on instance cloudinfra-idp-1 in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [20:37:28] (PuppetAgentStaleLastRun) firing: (8) Last Puppet run was over 24 hours ago on instance cloud-cumin-04 in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [20:41:11] 10Toolforge: [buildservice] "failed to create fsnotify watcher: too many open files" and "unable to open destination: open /tekton/home/.docker/config.json: permission denied" - https://phabricator.wikimedia.org/T361519 (10bd808) 03NEW [20:41:51] 10Toolforge (Software install/update), 13Patch-For-Review: Install php-yaml in Toolforge images - https://phabricator.wikimedia.org/T361457#9677722 (10bd808) p:05Triage→03Medium [20:43:57] 06cloud-services-team, 10Toolforge (Software install/update): Consider adding `kubectl`, `webservice`, and `toolforge` binaries to shell container images - https://phabricator.wikimedia.org/T360818#9677727 (10bd808) p:05Triage→03Medium [20:47:01] 06cloud-services-team, 10Toolforge (Software install/update): Consider adding `kubectl`, `webservice`, and `toolforge` binaries to shell container images - https://phabricator.wikimedia.org/T360818#9677735 (10bd808) Dropping this into the "needs discussion" column for #cloud-services-team as a potential soluti... [20:47:20] 06cloud-services-team, 10Toolforge (Software install/update): Missing Perl packages on dev.toolforge.org for anomiebot workflows - https://phabricator.wikimedia.org/T360488#9677736 (10bd808) p:05Triage→03High [20:48:20] 06cloud-services-team, 10Toolforge (Software install/update): Missing Perl packages on dev.toolforge.org for anomiebot workflows - https://phabricator.wikimedia.org/T360488#9677739 (10bd808) Dropping this into the "needs discussion" column for #cloud-services-team as a blocker to decommissioning the remaining... [21:08:05] 10Wikibugs: Wikibugs testing task - https://phabricator.wikimedia.org/T90594#9677768 (10bd808) test [21:11:12] 10Wikibugs: Wikibugs testing task - https://phabricator.wikimedia.org/T90594#9677780 (10bd808) test [21:18:41] (CloudVPSDesignateLeaks) firing: (5) Detected 3 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:52:07] (03PS1) 10Krinkle: frontend: Factor out Codesearch::getHoundApi() [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1016019 [22:52:07] (03PS1) 10Krinkle: frontend: Factor out getWithSetCallback() [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1016020 [22:52:07] (03PS1) 10Krinkle: frontend: Implement "action=excludes" view [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1016021 [22:53:48] (03CR) 10Krinkle: [C:03+2] frontend: Factor out Codesearch::getHoundApi() [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1016019 (owner: 10Krinkle) [22:54:36] (03CR) 10Krinkle: [C:03+2] frontend: Factor out getWithSetCallback() [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1016020 (owner: 10Krinkle) [22:54:55] (03Merged) 10jenkins-bot: frontend: Factor out Codesearch::getHoundApi() [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1016019 (owner: 10Krinkle) [22:55:27] (03Merged) 10jenkins-bot: frontend: Factor out getWithSetCallback() [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1016020 (owner: 10Krinkle) [22:57:14] (03PS2) 10Krinkle: frontend: Implement "action=excludes" view [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1016021 [22:57:43] (03CR) 10Krinkle: [C:03+2] frontend: Implement "action=excludes" view [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1016021 (owner: 10Krinkle) [22:58:38] (03Merged) 10jenkins-bot: frontend: Implement "action=excludes" view [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1016021 (owner: 10Krinkle)