[01:24:11] 10tool-wscontest: tool stuck and not updating - https://phabricator.wikimedia.org/T326111#10451291 (10Samwilson) 05Open→03Resolved a:03Samwilson The updating of the scores should be working better now. The main issue here, that the scoring system isn't counting the right things, is tracked in T325161. [01:25:26] 10tool-wscontest: Wikisource Contests tool does not fetch the stats - https://phabricator.wikimedia.org/T340219#10451295 (10Samwilson) 05Open→03Resolved a:03Samwilson The scoring updating system has been improved, so things should be working better now. Please open a new task if you notice any issues. [01:26:37] 10tool-wscontest: WS Contest has stopped updating its score - https://phabricator.wikimedia.org/T360749#10451299 (10Samwilson) 05Open→03Resolved a:03Samwilson Yep, things should be improved now. [01:28:12] 10tool-wscontest: Curl error setting certificate verify locations - https://phabricator.wikimedia.org/T222855#10451302 (10Samwilson) 05Open→03Invalid These errors are no longer occurring (we didn't change anything about the tool). [01:43:51] 10tool-wscontest, 07Accessibility, 07Voice & Tone: [[Wikimedia:Wscontest-click-here-link/en]] accessibility issue - https://phabricator.wikimedia.org/T367634#10451308 (10Samwilson) a:03Samwilson PR: https://github.com/wikisource/wscontest/pull/79 This changes to the following wording: {F58173164} [01:51:10] 10tool-wscontest: Add health-check-script for scores command runner - https://phabricator.wikimedia.org/T383304#10451315 (10Samwilson) a:03Samwilson [01:51:39] 10tool-wscontest: Add button to trigger manual run of scoring - https://phabricator.wikimedia.org/T343418#10451316 (10Samwilson) 05Open→03Stalled With the continuous processing introduced in T383304 this may no longer be needed. [02:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:41:33] 10VPS-Projects: Cannot create web proxy for matrix project - https://phabricator.wikimedia.org/T383511 (10MarkAHershberger) 03NEW [05:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:16:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-5:30000 has failed probes (http_admin_beta_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:31:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-5:30000 has failed probes (http_admin_beta_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:53:04] 06cloud-services-team, 10Toolforge: jobs-api: Indicate when a job is too big to be scheduled - https://phabricator.wikimedia.org/T383515 (10taavi) 03NEW [09:56:42] 10VPS-Projects: Cannot create web proxy for matrix project - https://phabricator.wikimedia.org/T383511#10451495 (10taavi) The `keycloak` instance is missing a security group to allow incoming traffic to port 8080. (The default group allows all project-local traffic, but traffic from the proxy needs a specific ru... [09:58:28] FIRING: InstanceDown: Project tools instance tools-prometheus-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [10:23:28] RESOLVED: InstanceDown: Project tools instance tools-prometheus-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [10:25:13] 06cloud-services-team, 10Toolforge: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.32 - https://phabricator.wikimedia.org/T379047#10451499 (10taavi) [10:25:15] 06cloud-services-team, 10Toolforge: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.31 - https://phabricator.wikimedia.org/T372697#10451500 (10taavi) [10:27:40] 06cloud-services-team, 10Toolforge: Upgrade ingress-nginx to v1.12.0+ - https://phabricator.wikimedia.org/T383516 (10taavi) 03NEW [10:28:41] 06cloud-services-team, 10Cloud-VPS: [wmcs-cookbooks] Use OpenStack APIs instead of using the CLIs as novaadmin - https://phabricator.wikimedia.org/T383517 (10taavi) 03NEW [11:30:18] 10tool-wscontest, 07good first task: Add sortable column for WSContest contest page - https://phabricator.wikimedia.org/T331509#10451541 (10Samwilson) a:05Sohamdas07→03Samwilson I've brought the old PR up to date and created a new PR for it: https://github.com/wikisource/wscontest/pull/80 [11:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:41:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-5:30000 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#toolsbeta-test-k8s-haproxy-5:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:46:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-5:30000 has failed probes (http_admin_beta_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:36:35] 06cloud-services-team, 10Toolforge: Toolforge jobs: increased exit code 137 rate since 2024-12-14 - https://phabricator.wikimedia.org/T382865#10451667 (10JJMC89) [17:37:54] 06cloud-services-team, 10Toolforge: [jobs-emailer] duplicate failure emails - https://phabricator.wikimedia.org/T382866#10451668 (10JJMC89) [17:44:25] 10VPS-Projects: Cannot create web proxy for matrix project - https://phabricator.wikimedia.org/T383511#10451672 (10MarkAHershberger) Isn't that this ingress rule? {F58177997} [17:54:10] 10VPS-Projects: Cannot create web proxy for matrix project - https://phabricator.wikimedia.org/T383511#10451674 (10MarkAHershberger) Nevermind. I missed the next paragraph: > You may also have to apply this new (or existing) security group to the instance you want to make available: navigate to "Instances" (in... [17:54:26] 10VPS-Projects: Cannot create web proxy for matrix project - https://phabricator.wikimedia.org/T383511#10451675 (10MarkAHershberger) 05Open→03Resolved a:03MarkAHershberger [18:50:41] FIRING: CloudVPSDesignateLeaks: Detected 3 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:57:02] 06cloud-services-team, 10Toolforge: Toolforge jobs: increased exit code 137 rate since 2024-12-14 - https://phabricator.wikimedia.org/T382865#10451681 (10JJMC89) [19:15:49] 10Tool-Global-user-contributions: Rollback edits are no longer counts as a contribution, although they are visible in the contribution history - https://phabricator.wikimedia.org/T383523 (10Nurtenge) 03NEW [19:17:20] 06cloud-services-team, 10Toolforge: [jobs-emailer] duplicate failure emails - https://phabricator.wikimedia.org/T382866#10451696 (10JJMC89) [19:20:21] 10Tool-Global-user-contributions: Rollback edits are no longer counts as a contribution, although they are visible in the contribution history - https://phabricator.wikimedia.org/T383523#10451709 (10Umherirrender) →14Duplicate dup:03T382592 [22:30:29] FIRING: InstanceDown: Project tools instance tools-k8s-worker-nfs-7 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [22:35:29] RESOLVED: InstanceDown: Project tools instance tools-k8s-worker-nfs-7 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [22:50:41] FIRING: CloudVPSDesignateLeaks: Detected 3 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:21:26] 06cloud-services-team, 10Toolforge: Toolforge jobs: increased exit code 137 rate since 2024-12-14 - https://phabricator.wikimedia.org/T382865#10451810 (10JJMC89)