[00:14:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [00:34:25] 10Wikibugs, 15User-bd808: wikibugs test bug part II - https://phabricator.wikimedia.org/T90594#9593763 (10bd808) test [00:47:56] (ProbeDown) firing: (2) Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:52:56] (ProbeDown) resolved: (2) Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:22:59] 10Wikibugs, 15User-bd808: wikibugs test bug part II - https://phabricator.wikimedia.org/T90594#9593770 (10bd808) test [01:30:54] 10Wikibugs, 15User-bd808: bd808's big pile of refactoring ideas - https://phabricator.wikimedia.org/T357851#9593771 (10bd808) The test deploy in the wikibugs-testing tool's namespace seems to be working now, including with a znc between the bot and libra.chat. I'm going to leave it running for a while before I... [01:35:12] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9593772 (10MBH) @dcaro I'm sorry, but how to use tools you transferred? * https://mbh.toolforge.org/cgi-bin/thanks-stats.html - this is raw html page, script doesn't run * htt... [01:36:45] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9593773 (10MBH) Also, webservice started by default with "web: php7.4", how to run it with python image? [01:41:06] 10Wikibugs, 10Quota-requests: Request increased quota for wikibugs-testing Toolforge tool - https://phabricator.wikimedia.org/T358968 (10bd808) [01:42:50] 10Wikibugs, 10Quota-requests: Request increased quota for wikibugs-testing Toolforge tool - https://phabricator.wikimedia.org/T358968#9593784 (10bd808) [01:58:56] (ProbeDown) firing: (2) Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:03:56] (ProbeDown) resolved: (2) Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:35:28] (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance tools-sgegrid-shadow in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [02:55:56] (ProbeDown) firing: (2) Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:05:56] (ProbeDown) resolved: (2) Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:14:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [03:32:28] (03CR) 10BryanDavis: [C: 03+1] "I guess I should submit an adoption request. It's been more than a year since I wrote (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:55:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:17:56] (ProbeDown) firing: Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-3:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:22:56] (ProbeDown) firing: (2) Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:27:56] (ProbeDown) resolved: (2) Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:38:56] (ProbeDown) firing: (2) Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:40:28] (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance tools-sgegrid-shadow in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [05:40:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:43:56] (ProbeDown) resolved: (2) Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:45:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:50:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:14:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [06:15:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:20:41] (CloudVPSDesignateLeaks) firing: (4) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:25:41] (CloudVPSDesignateLeaks) firing: (4) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:30:41] (CloudVPSDesignateLeaks) resolved: (4) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:53:56] (ProbeDown) firing: Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-3:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:58:56] (ProbeDown) resolved: (2) Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:53:56] (ProbeDown) firing: (2) Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:03:56] (ProbeDown) resolved: (2) Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:40:28] (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance tools-sgegrid-shadow in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [09:14:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [09:23:56] (ProbeDown) firing: Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-3:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:28:56] (ProbeDown) resolved: (2) Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:47:56] 10Wikibugs, 10Quota-requests: Request increased quota for wikibugs-testing Toolforge tool - https://phabricator.wikimedia.org/T358968#9593894 (10taavi) +1 [09:59:34] (DiskSpace) firing: Disk space cloudbackup1004:9100:/ 5.98% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [10:14:34] (DiskSpace) resolved: Disk space cloudbackup1004:9100:/ 5.978% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [10:45:50] (ProbeDown) firing: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:50:50] (ProbeDown) resolved: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:53:18] 10Toolforge, 07Kubernetes: kubectl is quite slow the “first time” per user account - https://phabricator.wikimedia.org/T358976 (10LucasWerkmeister) [10:56:20] (ProbeDown) firing: (3) Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:01:20] (ProbeDown) resolved: (3) Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:04:37] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9593942 (10dcaro) >>! In T319883#9593354, @MBH wrote: > @dcaro Oookay, but why after executing two of your commands by me, a "cgi-bin" folder was empty? The issue were permis... [11:09:51] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9593945 (10dcaro) yep, it seems that the proxy still using the grid backend: ` root@tools-proxy-06:~# grep thanks-stats /var/log/nginx/access.log ... mbh.toolforge.org 172.16.... [11:15:30] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9593952 (10dcaro) Cleared the entry on redis manually: ` root@tools-proxy-06:~# redis-cli ... 14) "prefix:mbh" ... 24) "redirect:mbh" ... 127.0.0.1:6379> del prefix:mbh (inte... [11:18:14] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9593953 (10MBH) Okay. Now https://mbh.toolforge.org/cgi-bin/likes.cgi?user=MBH&wiki=ru.wikipedia responds with `Message: No such CGI script ('/cgi-bin/likes.cgi').` What scri... [11:18:58] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9593954 (10dcaro) I think that the grid webservice was not stopped or something, as it was still running, just stopped it manually too: ` tools.mbh@tools-sgebastion-10:~$ qsta... [11:26:45] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9593956 (10dcaro) >>! In T319883#9593953, @MBH wrote: > Okay. Now https://mbh.toolforge.org/cgi-bin/likes.cgi?user=MBH&wiki=ru.wikipedia responds with `Message: No such CGI sc... [11:40:28] (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance tools-sgegrid-shadow in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [11:41:29] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9593971 (10MBH) //I don't see a likes.cs here// - because you moved it as `thanks-stats.cs`. I just used more understandable name for tool when I publish its code. [11:42:25] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9593972 (10dcaro) hmm, I think that the likes.cgi is the thanks-stats one? If so, it should point to `
`, trying manually to use the "fixed" url end... [11:44:20] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9593973 (10dcaro) >>! In T319883#9593971, @MBH wrote: > //I don't see a likes.cs here// - because you moved it as `thanks-stats.cs`. I just used more understandable name for t... [12:13:56] (ProbeDown) firing: (2) Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:14:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [12:18:56] (ProbeDown) resolved: (2) Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:36:08] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9594014 (10MBH) Okay, I redacted and renamed project files, including path to password file https://github.com/Saisengen/wikibots/commit/8b1539f87f5d3a67753746854b8e8b8dab6279... [12:38:39] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9594015 (10dcaro) I think you missed updating the solutions file: https://github.com/Saisengen/wikibots/blob/main/web-services/web-services.sln#L6 [12:41:39] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9594032 (10MBH) Thanks. This IDs - where do I get them for other bots? ` Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "likes", "likes\likes.csproj", "{FD320275-1E4A-46D... [12:46:51] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9594033 (10MBH) I edited solutions file, run building again, "cgi-bin" folder is still unchanged. [12:47:02] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9594034 (10dcaro) >>! In T319883#9594032, @MBH wrote: > Thanks. This IDs - where do I get them for other bots? > ` > Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "likes... [12:48:50] (ProbeDown) firing: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:50:34] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9594038 (10dcaro) Just tried locally and it worked with your latest build: ` dcaro@urcuchillay$ podman run --userns keep-id --rm -ti --volume $PWD/kk:/toolhome:rw,z --env TOOL... [12:51:04] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9594039 (10dcaro) it's there on your tool too: ` tools.mbh@tools-sgebastion-10:~$ ls -la public_html/cgi-bin/likes -rwxr-xr-x 1 tools.mbh tools.mbh 72520 Mar 3 12:46 public_h... [12:52:00] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9594040 (10dcaro) and it seems to work: https://mbh.toolforge.org/cgi-bin/likes [12:53:50] (ProbeDown) resolved: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:01:59] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9594043 (10MBH) Thank you very much. Is there no way to automatically remove deleted and renamed tool files from "cgi-bin" folder, I have to delete them manually? And another... [13:14:43] 10Tool-Phabricator-bug-status: Move phabricator-bug-status to kubernetes - https://phabricator.wikimedia.org/T142237#9594047 (10Xover) Matt hasn't edited since 2019 and no longer works for the WMF so is unlikely to show up and fix the tool. But if anybody feels up for usurping it, the code is at https://gerrit.... [13:19:09] 10Tool-Phabricator-bug-status: Move phabricator-bug-status to kubernetes - https://phabricator.wikimedia.org/T142237#9594054 (10taavi) 05Open→03Resolved https://k8s-status.toolforge.org/namespaces/tool-phabricator-bug-status/ shows the tool running on the Kubernetes backend. [13:21:02] 10Toolforge: Alert when admin managed pods are having issues - https://phabricator.wikimedia.org/T358909#9594056 (10taavi) [13:21:56] (ProbeDown) firing: Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-3:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:26:56] (ProbeDown) resolved: (2) Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:35:10] 10Tool-Phabricator-bug-status: Usurp and move phabricator-bug-status to the Toolforge Jobs Framework - https://phabricator.wikimedia.org/T142237#9594063 (10Xover) 05Resolved→03Open [13:36:45] 10Tool-Phabricator-bug-status: Usurp and move phabricator-bug-status to the Toolforge Jobs Framework - https://phabricator.wikimedia.org/T142237#9594067 (10Xover) >>! In T142237#9594054, @taavi wrote: > https://k8s-status.toolforge.org/namespaces/tool-phabricator-bug-status/ shows the tool running on the Kuberne... [13:52:50] (ProbeDown) firing: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:57:50] (ProbeDown) resolved: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:21:00] 10Wikibugs, 15User-bd808: bd808's big pile of refactoring ideas - https://phabricator.wikimedia.org/T357851#9594089 (10bd808) Clone the repo and create a venv for use with the python3.9 runtime container: `lang=shell-session $ git clone https://gitlab.wikimedia.org/toolforge-repos/wikibugs2.git $ webservice --... [14:33:37] 10VPS-project-Codesearch: Index labs/toollabs - https://phabricator.wikimedia.org/T358983 (10Bugreporter) [14:38:09] 10PAWS: Add a simple script to connect to a replica database - https://phabricator.wikimedia.org/T358984 (10Bugreporter) [14:40:28] (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance tools-sgegrid-shadow in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [14:51:56] 10Wikibugs, 15User-bd808: wikibugs test bug part II - https://phabricator.wikimedia.org/T90594#9594133 (10bd808) does the older redis2irc still work? [14:58:47] 10Wikibugs, 15User-bd808: bd808's big pile of refactoring ideas - https://phabricator.wikimedia.org/T357851#9594145 (10bd808) >>! In T357851#9594130, @bd808 wrote: > The pods seem to be running as expected, but the irc bot is not picking up messages from the redis queue. Rolling back and then will inspect conf... [15:14:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [15:19:09] 10Wikibugs, 15User-bd808: bd808's big pile of refactoring ideas - https://phabricator.wikimedia.org/T357851#9594166 (10bd808) ok. new plan: I've set a secret queue name for the new code to use via `toolforge envvars create REDIS_QUEUE_NAME`. Now I will switch the gerrit and phorge feeds to the new code and tha... [15:22:26] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9594171 (10dcaro) >>! In T319883#9594043, @MBH wrote: > Thank you very much. Is there no way to automatically remove deleted and renamed tool files from "cgi-bin" folder, I ha... [15:27:43] (03PS1) 10BryanDavis: [DONT MERGE] Testing wikibugs [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/1008016 [15:31:48] 10Wikibugs, 15User-bd808: bd808's big pile of refactoring ideas - https://phabricator.wikimedia.org/T357851#9594176 (10bd808) `lang=shell-session $ toolforge jobs delete grrrrit $ toolforge jobs load --job gerrit wikibugs2/toolforge-jobs.yaml $ kubectl logs --all-containers=true --ignore-errors --since=10m -f... [15:37:33] (03CR) 10BryanDavis: [C: 04-1] "Test" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/1008016 (owner: 10BryanDavis) [15:37:46] 10Wikibugs, 15User-bd808: wikibugs test bug part II - https://phabricator.wikimedia.org/T90594#9594178 (10bd808) 05Stalled→03In progress test [15:37:54] 10Wikibugs: wikibugs test bug - https://phabricator.wikimedia.org/T1152#9594180 (10bd808) [15:38:10] 10Wikibugs, 15User-bd808: bd808's big pile of refactoring ideas - https://phabricator.wikimedia.org/T357851#9594181 (10bd808) `lang=shell-session $ toolforge jobs delete wikibugs-phab $ toolforge jobs load --job phorge wikibugs2/toolforge-jobs.yaml $ kubectl logs --all-containers=true --ignore-errors --since=1... [15:38:50] (ProbeDown) firing: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:42:37] (03CR) 10BryanDavis: [C: 04-2] "Is -2 wikispeak for "wow, what a bad idea?" or "I'm being very careful."? Depends on context I'm pretty sure." [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/1008016 (owner: 10BryanDavis) [15:43:30] 10Wikibugs, 15User-bd808: bd808's big pile of refactoring ideas - https://phabricator.wikimedia.org/T357851#9594194 (10bd808) `lang=shell-session $ toolforge jobs delete redis2irc $ toolforge jobs load --job znc wikibugs2/toolforge-jobs.yaml $ toolforge jobs load --job irc wikibugs2/toolforge-jobs.yaml $ kubec... [15:43:50] (ProbeDown) resolved: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:48:44] 10Wikibugs, 15User-bd808: wikibugs having a hard time staying connected to libera.chat IRC network - https://phabricator.wikimedia.org/T357729#9594236 (10bd808) The bot is now running with a znc instance between it and libera.chat. The hope here is that znc has more robust algorithms for detecting connection i... [16:33:56] (ProbeDown) firing: Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-3:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:38:56] (ProbeDown) resolved: (2) Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:45:14] 10Toolforge: `toolforge webservice TYPE shell -- something` does not pass extra cli arguments like `webservice TYPE shell -- something` does - https://phabricator.wikimedia.org/T358999 (10bd808) [16:50:26] 10Toolforge: `toolforge webservice TYPE shell -- something` does not pass extra cli arguments like `webservice TYPE shell -- something` does - https://phabricator.wikimedia.org/T358999#9594374 (10bd808) At first glance the issue is that `--` is not treated as a end of arguments marker by the `toolforge webservic... [17:40:28] (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance tools-sgegrid-shadow in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [17:49:17] 10Wikibugs, 15User-bd808: wikibugs only shows milestone name without parent project name - https://phabricator.wikimedia.org/T358653#9594484 (10bd808) 05Open→03In progress a:03bd808 [18:05:19] 10Wikibugs, 13Patch-For-Review, 15User-bd808: wikibugs only shows milestone name without parent project name - https://phabricator.wikimedia.org/T358653#9594490 (10CodeReviewBot) bd808 opened https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/8 phorge: Construct fully qualified names fo... [18:07:56] (ProbeDown) firing: (2) Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:12:56] (ProbeDown) resolved: (2) Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:14:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [19:23:56] (ProbeDown) firing: (2) Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:28:56] (ProbeDown) resolved: (2) Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:37:28] (PuppetCertificateAboutToExpire) firing: Puppet CA certificate Puppet CA: cloudinfra-internal-puppetmaster01.cloudinfra.eqiad.wmflabs is about to expire in 27d 23h 58m 11s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [20:40:28] (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance tools-sgegrid-shadow in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [21:01:56] (ProbeDown) firing: (2) Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:06:56] (ProbeDown) resolved: (2) Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:10:04] 10Striker, 10Wikibugs, 10FY2023/2024-Q3-Q4, 15User-bd808: wikibugs test bug part II - https://phabricator.wikimedia.org/T90594#9594582 (10bd808) Adding a milestone project (#wmcs-current) and a subproject (#striker) to capture some debug data. [21:12:37] 10Wikibugs, 15User-bd808: wikibugs test bug part II - https://phabricator.wikimedia.org/T90594#9594585 (10bd808) [21:14:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [22:34:01] 10Wikibugs, 15User-bd808: bd808's big pile of refactoring ideas - https://phabricator.wikimedia.org/T357851#9594608 (10bd808) 05In progress→03Resolved Docs have been updated at https://www.mediawiki.org/wiki/Wikibugs for the new deployment. I think we can call this {{done}}. [22:34:03] 10Wikibugs, 15User-bd808: Re-enable git hook handler to update git clone - https://phabricator.wikimedia.org/T358967#9594610 (10bd808) [22:34:28] (03PS2) 10BryanDavis: [DONT MERGE] Testing wikibugs [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/1008016 (https://phabricator.wikimedia.org/T90594) [22:34:36] (03CR) 10CI reject: [V: 04-1] [DONT MERGE] Testing wikibugs [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/1008016 (https://phabricator.wikimedia.org/T90594) (owner: 10BryanDavis) [22:37:50] (ProbeDown) firing: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:39:38] 10Striker, 10Wikibugs, 10FY2023/2024-Q3-Q4, 13Patch-For-Review, 15User-bd808: wikibugs test bug part II - https://phabricator.wikimedia.org/T90594#9594613 (10bd808) Putting milestone and subproject tags back because, fun fact, when processing a Phorge event we get the core task information including tags... [22:42:50] (ProbeDown) firing: (3) Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:47:50] (ProbeDown) resolved: (3) Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:28:39] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9594629 (10MBH) > They are set/generated by `dotnet sln add web-service//.csproj` ` tools.mbh@tools-sgebastion-10:~$ dotnet sln add web-services/u... [23:34:14] 10Wikibugs, 15User-bd808: wikibugs only shows milestone name without parent project name - https://phabricator.wikimedia.org/T358653#9594631 (10CodeReviewBot) bd808 merged https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/8 phorge: Construct fully qualified names for milestone projects [23:39:17] 10Striker, 10Wikibugs, 10cloud-services-team (FY2023/2024-Q3-Q4), 13Patch-For-Review, 15User-bd808: wikibugs test bug part II - https://phabricator.wikimedia.org/T90594#9594644 (10bd808) Is {T358653} fixed? [23:40:28] (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance tools-sgegrid-shadow in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [23:40:50] 10Wikibugs, 15User-bd808: wikibugs only shows milestone name without parent project name - https://phabricator.wikimedia.org/T358653#9594646 (10bd808) 05In progress→03Resolved `lang=irc [23:39] < wikibugs> Striker, Wikibugs, cloud-services-team (FY2023/2024-Q3-Q4), Patch-For-Review, User-bd808: wikibugs t... [23:41:32] 10Wikibugs: wikibugs test bug part II - https://phabricator.wikimedia.org/T90594#9594648 (10bd808) [23:42:28] (PuppetCertificateAboutToExpire) firing: Puppet CA certificate Puppet CA: cloudinfra-internal-puppetmaster01.cloudinfra.eqiad.wmflabs is about to expire in 27d 20h 54m 11s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [23:46:17] 10Wikibugs, 15User-bd808: Re-enable git hook handler to update git clone - https://phabricator.wikimedia.org/T358967#9594650 (10bd808) 05Stalled→03Resolved Hook handler was re-enabled and tested with the merge of {6e10c374}: https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/jobs/218147 [23:53:32] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9594655 (10MBH) How to configure building process to not only files in "web-services" folder on GitHub are builded, but also "cluster-analysis" folder are built too? https://... [23:56:01] 10Toolforge (Quota-requests), 10Wikibugs, 13Patch-For-Review, 15User-bd808: Request increased quota for wikibugs-testing Toolforge tool - https://phabricator.wikimedia.org/T358968#9594656 (10CodeReviewBot) bd808 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/211... [23:56:51] 10Toolforge (Quota-requests), 10Wikibugs, 13Patch-For-Review, 15User-bd808: Request increased quota for wikibugs-testing Toolforge tool - https://phabricator.wikimedia.org/T358968#9594660 (10bd808) 05Open→03In progress a:03bd808