[00:27:20] 10PAWS: Update pywikibot to 9.0.0 - https://phabricator.wikimedia.org/T359673#9616875 (10Xqt) >>! In T359673#9616516, @taavi wrote: > Dupe of {T359616}... you know the automation in LibUp for filing these tasks works again, right? The last few releases it created a task after weeks or months. This time it was a... [00:35:28] (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 1 deleted instances on metricsinfra-puppetmaster-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [01:14:36] 05Grid-Engine-to-K8s-Migration: Migrate multichill from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319912#9617031 (10AntiCompositeNumber) The sabotaging of grid jobs happens on Thursday. [01:25:56] 05Grid-Engine-to-K8s-Migration, 06Growth-Team, 10Community-Tech (CommTech-Kanban): Migrate ERANBOT project off of Grid Engine - https://phabricator.wikimedia.org/T306888#9617068 (10MusikAnimal) I ended up installing packages manually inside `webservice python2 shell`. Things were looking promising, but ultim... [02:58:53] 05Grid-Engine-to-K8s-Migration, 06Growth-Team, 10Community-Tech (CommTech-Kanban): Migrate ERANBOT project off of Grid Engine - https://phabricator.wikimedia.org/T306888#9617210 (10JJMC89) `PYTHONPATH` shenanigans `lang=shell-session tools.eranbot@tools-sgebastion-11:~$ webservice python2 shell tools.eranbo... [03:35:28] (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 1 deleted instances on metricsinfra-puppetmaster-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [03:52:23] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9617239 (10MBH) Thank you very much, but it doesn't solve some of errors. I did a relative path replacement you suggested in two tools * https://github.com/Saisengen/wikibots... [05:39:46] (03CR) 10BryanDavis: [C: 04-2] "> test asyncssh in local dev" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/1008016 (https://phabricator.wikimedia.org/T90594) (owner: 10BryanDavis) [05:40:53] (03CR) 10BryanDavis: [C: 04-2] "again" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/1008016 (https://phabricator.wikimedia.org/T90594) (owner: 10BryanDavis) [05:43:49] (03CR) 10BryanDavis: [C: 04-2] "once more with feeling" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/1008016 (https://phabricator.wikimedia.org/T90594) (owner: 10BryanDavis) [05:44:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:52:57] (03CR) 10Eugene233: "recheck" [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1008917 (owner: 10Josefanthony) [05:54:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:35:28] (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 1 deleted instances on metricsinfra-puppetmaster-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [06:43:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:53:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:20:00] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [09:35:28] (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 1 deleted instances on metricsinfra-puppetmaster-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [10:19:59] 14Toolforge (Toolforge iteration 06), 13Patch-For-Review: Support probes in kubernetes webservices - https://phabricator.wikimedia.org/T341919#9617675 (10Dvorapa) Could someone point me to how to make the probe not appear in access.log for perl5.36 webservice? [10:36:46] 05Grid-Engine-to-K8s-Migration: Migrate ganfilter from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357554#9617676 (10coldchrist) Thanks, @JJMC89, I thought I'd responded to your post yesterday but apparently not. I removed the PYTHONPATH setting from my bash profile, and ch... [10:57:14] (03PS1) 10Majavah: tools: Restore original membership pagination [labs/striker] - 10https://gerrit.wikimedia.org/r/1009827 [12:32:00] (03Merged) 10jenkins-bot: tools: Restore original membership pagination [labs/striker] - 10https://gerrit.wikimedia.org/r/1009827 (owner: 10Majavah) [12:32:52] 14Toolforge (Toolforge iteration 06), 13Patch-For-Review: Support probes in kubernetes webservices - https://phabricator.wikimedia.org/T341919#9617698 (10LucasWerkmeister) >>! In T341919#9617675, @Dvorapa wrote: > Could someone point me to how to make the probe not appear in access.log for perl5.36 webservice?... [12:40:28] (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 1 deleted instances on metricsinfra-puppetmaster-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [12:56:50] (ProbeDown) firing: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:01:50] (ProbeDown) resolved: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:20:01] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [14:49:07] 05Grid-Engine-to-K8s-Migration: Migrate ganfilter from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357554#9617793 (10dcaro) >>! In T357554#9617676, @coldchrist wrote: > Thanks, @JJMC89, I thought I'd responded to your post yesterday but apparently not. I removed the PYTHONP... [15:06:07] 05Grid-Engine-to-K8s-Migration: Migrate ganfilter from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357554#9617799 (10coldchrist) See https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Bot_that_maintains_GA_nominations_page_is_down -- @SD0001 offered to be a mai... [15:06:25] 10PAWS: Increase paws nodes for outreachy demand - https://phabricator.wikimedia.org/T359747 (10rook) 03NEW [15:40:28] (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 1 deleted instances on metricsinfra-puppetmaster-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [16:04:46] 10PAWS: Increase paws nodes for outreachy demand - https://phabricator.wikimedia.org/T359747#9617831 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/388 [16:04:58] vivian-rook opened https://github.com/toolforge/paws/pull/388 [16:05:12] 05Grid-Engine-to-K8s-Migration: Migrate ganfilter from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357554#9617832 (10SD0001) The version of the code on toolforge at the time [[https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#c-SD0001-20240309121600-Mike_Chris... [16:05:58] 10PAWS: Increase paws nodes for outreachy demand - https://phabricator.wikimedia.org/T359747#9617833 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/388 [16:06:12] vivian-rook closed https://github.com/toolforge/paws/pull/388 [16:06:19] 10PAWS: Increase paws nodes for outreachy demand - https://phabricator.wikimedia.org/T359747#9617835 (10rook) 05Open→03Resolved [16:18:07] 05Grid-Engine-to-K8s-Migration: Migrate ganfilter from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357554#9617838 (10coldchrist) OK, I've reverted to the code without dcaro's suggested changes -- that should be what you ran successfully. Re the venv command per your comment... [16:33:08] 05Grid-Engine-to-K8s-Migration: Migrate ganfilter from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357554#9617845 (10SD0001) That should be `source ~/www/python/venv/bin/activate`, sorry. [16:34:22] (HAProxyBackendUnavailable) firing: HAProxy service nova-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [16:44:22] (HAProxyBackendUnavailable) resolved: HAProxy service nova-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [16:56:31] (03CR) 10BryanDavis: [C: 04-2] "test" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/1008016 (https://phabricator.wikimedia.org/T90594) (owner: 10BryanDavis) [17:03:50] (ProbeDown) firing: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:08:50] (ProbeDown) resolved: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:20:15] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [17:32:28] 05Grid-Engine-to-K8s-Migration: Migrate ganfilter from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T357554#9617905 (10coldchrist) The bot is now running under the toolforge command suggested at the VPT thread. I'll work on some other clean up issues such as using the env var... [18:00:01] (OpenstackAPIResponse) firing: (2) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [18:11:37] (03PS2) 10AgnesAbah: Bug:T357376 Remove unused/not-required imports from _init_.py [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1006202 [18:11:39] (03PS2) 10AgnesAbah: Bug:T357376 Remove unused/not-required imports from _init_.py [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1007683 [18:11:41] (03PS1) 10AgnesAbah: Bug:T357376 Remove unused/not-required imports from _init_.py [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1009399 [18:13:18] (03CR) 10AgnesAbah: "check i have removed the lines" [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1009399 (owner: 10AgnesAbah) [18:40:28] (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 1 deleted instances on metricsinfra-puppetmaster-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [18:43:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:52:59] 10Wikibugs: Rethink anti-flooding protections - https://phabricator.wikimedia.org/T359753 (10bd808) 03NEW [18:53:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:05:53] 10Wikibugs: Rethink anti-flooding protections - https://phabricator.wikimedia.org/T359753#9617952 (10bd808) p:05Triage→03High I am working to make the gerrit producer async which also means moving away from `wikibugs2.rqueue.RedisQueue`. The past discussions of flooding I have seen are related to Phorge bulk... [19:23:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:58:52] 10Toolforge, 06cloud-services-team, 13Patch-For-Review: Toolforge: Introduce grid-less bookworm based bastion hosts - https://phabricator.wikimedia.org/T314665#9617978 (10bd808) `lang=irc [19:47] < bd808> the what to keep question is I think the interesting one. Getting rid of lots of language runtime th... [20:19:36] (03PS1) 10AgnesAbah: Bug:T359300 Wrote docstrings for functions on the ISA tool on routes.py [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1009400 [20:21:02] (03CR) 10AgnesAbah: "Bug:T359300 Wrote docstrings for functions on the ISA tool on routes.py | https://gerrit.wikimedia.org/r/c/labs/tools/Isa/+/1009400" [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1009400 (owner: 10AgnesAbah) [20:50:31] (03CR) 10BryanDavis: [C: 04-2] "test" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/1008016 (https://phabricator.wikimedia.org/T90594) (owner: 10BryanDavis) [21:00:01] (OpenstackAPIResponse) firing: (2) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [21:12:20] (03CR) 10BryanDavis: [C: 04-2] "test" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/1008016 (https://phabricator.wikimedia.org/T90594) (owner: 10BryanDavis) [21:13:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:18:24] (03CR) 10BryanDavis: [C: 04-2] "test" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/1008016 (https://phabricator.wikimedia.org/T90594) (owner: 10BryanDavis) [21:20:15] 10Tool-wlh: Add an extra sort option - by article size - https://phabricator.wikimedia.org/T359756 (10The_Equalizer) 03NEW [21:23:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:24:23] 10Wikibugs: Rethink anti-flooding protections - https://phabricator.wikimedia.org/T359753#9618016 (10valhallasw) I think indeed the current system was primarily a "well this seems good enough" solution rather than taking a principled approach. I don't remember whether the current approach is a 'buffer' or 'dro... [21:36:22] (HAProxyBackendUnavailable) firing: HAProxy service nova-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [21:40:28] (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 1 deleted instances on metricsinfra-puppetmaster-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [21:44:24] (03PS1) 10AgnesAbah: Bug:T359300 Wrote docstrings for functions on the ISA tool on routes.py [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1009401 [21:51:19] (03PS1) 10AgnesAbah: Bug:T343438 has been corrected to up to six caption languages and one depicts language in routes.py [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1009402 [21:58:01] (03CR) 10AgnesAbah: ""More than on language" has been corrected to "up to six caption languages and one depicts language" to clarify the number of languages us" [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1009402 (owner: 10AgnesAbah) [22:16:52] 10Wikibugs: Rethink anti-flooding protections - https://phabricator.wikimedia.org/T359753#9618035 (10bd808) >>! In T359753#9618016, @valhallasw wrote: > I don't remember whether the current approach is a 'buffer' or 'drop' - there is value in just giving up at some point (allowing more recent events to be emitte... [23:19:00] 10Wikibugs, 13Patch-For-Review, 15User-bd808: Bot does not detect when ssh connection to Gerrit is interrupted - https://phabricator.wikimedia.org/T359096#9618044 (10CodeReviewBot) bd808 opened https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/14 gerrit: Use asyncssh [23:23:34] 10Wikibugs: wikibugs test bug - https://phabricator.wikimedia.org/T1152#9618053 (10bd808) [23:23:36] 10Wikibugs: Wikibugs testing task - https://phabricator.wikimedia.org/T90594#9618049 (10bd808) 05In progress→03Open p:05Triage→03Low [23:33:23] (03CR) 10BryanDavis: [C: 04-2] "Done" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/1008016 (https://phabricator.wikimedia.org/T90594) (owner: 10BryanDavis)