[01:59:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:04:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:09:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:14:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:24:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:29:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:34:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:36:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [04:41:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [04:54:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:59:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:04:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:09:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:24:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:12:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:22:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:29:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:34:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:31:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:36:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:41:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:46:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:02:31] 06cloud-services-team, 10PAWS, 06tools-platform-team: PAWS: restore nbgitpuller — removing it was never necessary - https://phabricator.wikimedia.org/T434973 (10Audiodude) 03NEW [10:06:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:09:37] 06cloud-services-team, 10PAWS, 06tools-platform-team: PAWS: restore nbgitpuller — removing it was never necessary - https://phabricator.wikimedia.org/T434973#12218383 (10Audiodude) [10:11:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:16:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:25:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:30:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [11:35:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:35:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [13:55:11] 10Striker, 06tools-platform-team: 500 Internal Server Error when trying to log in to toolsadmin - https://phabricator.wikimedia.org/T434904#12218449 (10Malyzolwikson) Same problem here. I've tried different browsers and computers. Problem persists. [14:55:41] 10Striker, 06tools-platform-team: 500 Internal Server Error when trying to log in to toolsadmin - https://phabricator.wikimedia.org/T434904#12218504 (10Andrew) This works for me, but the striker logs have quite a lot of this: ` @cee: {"@timestamp": "2026-08-15T14:46:42.230Z", "@version": "1", "message": "Int... [14:58:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:03:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:04:58] 10Striker, 06tools-platform-team: 500 Internal Server Error when trying to log in to toolsadmin - https://phabricator.wikimedia.org/T434904#12218521 (10Andrew) @FlammablePizza and @Malyzolwikson -- since this works for me, I'm thinking this issue is related to a set of particular accounts (maybe most accounts)... [15:07:42] 10Striker, 06tools-platform-team: 500 Internal Server Error when trying to log in to toolsadmin - https://phabricator.wikimedia.org/T434904#12218522 (10Andrew) Also: please comment if this is your first time logging into toolsadmin or if you have used it in the past. Thanks [15:10:26] 10Tool-wikinewsie: Wikinewsie portals flexibility and user-generated content - https://phabricator.wikimedia.org/T434980 (10Pharos) 03NEW [15:30:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:35:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:39:07] 10Striker, 06tools-platform-team: 500 Internal Server Error when trying to log in to toolsadmin - https://phabricator.wikimedia.org/T434904#12218550 (10Schiste) I had that same issue around 6am UTC with this account on tool hub specifically, but now it worked successfully. [15:40:29] 10Striker, 06tools-platform-team: 500 Internal Server Error when trying to log in to toolsadmin - https://phabricator.wikimedia.org/T434904#12218551 (10Schiste) I had the same issue this morning around 6am UTC, and just had it again around 17:39 CEST with my account schiste [15:59:15] 10Striker, 06tools-platform-team: 500 Internal Server Error when trying to log in to toolsadmin - https://phabricator.wikimedia.org/T434904#12218571 (10Andrew) It worked before everytime I needed. And I can ssh to toolforge and connect to Toolhub or Phabricator, it seems only Toolsadmin doesn't li... [16:14:33] 10Striker, 06tools-platform-team: 500 Internal Server Error when trying to log in to toolsadmin - https://phabricator.wikimedia.org/T434904#12218575 (10Andrew) Here is a more useful error log ` "@timestamp": "2026-08-15T15:39:36.930Z", "@version": "1", "message": "Internal Server Error: /auth/login/", "host":... [16:19:39] 10Striker, 06tools-platform-team: 500 Internal Server Error when trying to log in to toolsadmin - https://phabricator.wikimedia.org/T434904#12218576 (10Andrew) ` @cee: {"@timestamp": "2026-08-15T15:39:36.677Z", "@version": "1", "message": "\"POST /auth/login/ HTTP/1.1\" 500 6228", "host": "cloudweb1003", "path... [16:20:27] (03PS1) 10Perryprog: More dev environment adjustments for MW 1.43 [labs/striker] - 10https://gerrit.wikimedia.org/r/1326015 [16:23:02] 10Striker, 06tools-platform-team: 500 Internal Server Error when trying to log in to toolsadmin - https://phabricator.wikimedia.org/T434904#12218577 (10Andrew) Note that striker was redeployed 2 days ago, so this is likely an issue with code changes and not e.g. an ldap outage or ID overflow. [16:32:38] (03PS1) 10Majavah: labsauth: Fix accidentally shared object_classes list [labs/striker] - 10https://gerrit.wikimedia.org/r/1326016 (https://phabricator.wikimedia.org/T434684) [16:36:46] (03CR) 10Andrew Bogott: [C:03+1] labsauth: Fix accidentally shared object_classes list [labs/striker] - 10https://gerrit.wikimedia.org/r/1326016 (https://phabricator.wikimedia.org/T434684) (owner: 10Majavah) [16:39:19] (03CR) 10Andrew Bogott: [C:03+2] labsauth: Fix accidentally shared object_classes list [labs/striker] - 10https://gerrit.wikimedia.org/r/1326016 (https://phabricator.wikimedia.org/T434684) (owner: 10Majavah) [16:40:51] (03Merged) 10jenkins-bot: labsauth: Fix accidentally shared object_classes list [labs/striker] - 10https://gerrit.wikimedia.org/r/1326016 (https://phabricator.wikimedia.org/T434684) (owner: 10Majavah) [16:47:29] 10Striker, 06tools-platform-team: 500 Internal Server Error when trying to log in to toolsadmin - https://phabricator.wikimedia.org/T434904#12218586 (10taavi) →14Duplicate dup:03T434684 [16:47:31] 10Striker, 06tools-platform-team, 13Patch-For-Review: 500 internal error when accessing toolsadmin ssh-keys - https://phabricator.wikimedia.org/T434684#12218588 (10taavi) [17:07:14] 10Striker, 06tools-platform-team, 13Patch-For-Review: 500 internal error when accessing toolsadmin ssh-keys - https://phabricator.wikimedia.org/T434684#12218598 (10Andrew) 05Open→03Resolved a:03Andrew [17:12:58] (03CR) 10Majavah: [C:03+2] More dev environment adjustments for MW 1.43 [labs/striker] - 10https://gerrit.wikimedia.org/r/1326015 (owner: 10Perryprog) [17:14:30] (03Merged) 10jenkins-bot: More dev environment adjustments for MW 1.43 [labs/striker] - 10https://gerrit.wikimedia.org/r/1326015 (owner: 10Perryprog) [17:31:58] 10Striker, 06tools-platform-team: 500 internal error when accessing toolsadmin ssh-keys - https://phabricator.wikimedia.org/T434684#12218627 (10taavi) a:05Andrew→03taavi [17:57:39] (03PS1) 10Majavah: dev(docker): Update GitLab container to latest v19 [labs/striker] - 10https://gerrit.wikimedia.org/r/1326021 [18:40:32] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 10Ceph, and 3 others: cloudcephosd1044 boot issues - https://phabricator.wikimedia.org/T429267#12218644 (10VRiley-WMF) Received this from Dell 1. Drain Server power PowerEdge: How To Reset and Power Drain or Do a Virtual AC Cycle | Dell... [18:41:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [18:46:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [19:37:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:42:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:23:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:33:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:38:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:48:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:01:17] 10Tool-wd-act: Setup a standard secret management approach for local developers - https://phabricator.wikimedia.org/T434990 (10Tekneos) 03NEW [22:05:07] 10Tool-wd-act: Figure out how/if to integrate auth into standard python/flask/library auth facilities - https://phabricator.wikimedia.org/T431042#12218772 (10Tekneos) I've got a WIP branch here to do this: https://gitlab.wikimedia.org/toolforge-repos/wd-act/-/tree/T431042 It still needs to work through issues l... [22:29:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:34:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:40:16] (03open) 10tekneos: Simplify local development scripts to work correctly with gitignore, etc. #T434990 [toolforge-repos/wd-act] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-act/-/merge_requests/2 [22:40:57] (03approved) 10tekneos: Simplify local development scripts to work correctly with gitignore, etc. #T434990 [toolforge-repos/wd-act] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-act/-/merge_requests/2 [22:41:06] (03merge) 10tekneos: Simplify local development scripts to work correctly with gitignore, etc. #T434990 [toolforge-repos/wd-act] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-act/-/merge_requests/2 [22:44:29] 10Tool-wd-act, 13Patch-For-Review: Setup a standard secret management approach for local developers - https://phabricator.wikimedia.org/T434990#12218789 (10Tekneos) Added a script setup to work easily on local dev machines. Still need to update dev documentation and make sure the production config is ok. [22:47:44] 10Tool-wd-act, 13Patch-For-Review: Setup a standard secret management approach for local developers - https://phabricator.wikimedia.org/T434990#12218790 (10Tekneos) Docs updated here: https://wikitech.wikimedia.org/wiki/Tool:WikiDataAuthorityControlTool#Configure [23:05:50] 10Tool-wikinewsie: Wikinewsie portals flexibility and user-generated content - https://phabricator.wikimedia.org/T434980#12218800 (10Mikeyeak86) [23:05:52] 06cloud-services-team, 10PAWS, 06tools-platform-team: PAWS: restore nbgitpuller — removing it was never necessary - https://phabricator.wikimedia.org/T434973#12218804 (10Mikeyeak86) [23:51:54] 10Tool-wikinewsie: Wikinewsie portals flexibility and user-generated content - https://phabricator.wikimedia.org/T434980#12218813 (10Reedy) [23:51:56] 06cloud-services-team, 10PAWS, 06tools-platform-team: PAWS: restore nbgitpuller — removing it was never necessary - https://phabricator.wikimedia.org/T434973#12218817 (10Reedy) [23:52:34] 10Tool-ldap: Show SSH key fingerprints - https://phabricator.wikimedia.org/T434054#12218822 (10Reedy) p:05Unbreak!→03Triage