[00:16:28] (InstanceDown) firing: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:21:28] (InstanceDown) resolved: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [01:00:28] (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 1 deleted instances on metricsinfra-puppetmaster-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [01:03:03] 10Wikibugs: Investigate producing a code quality report for GitLab based on flake8 - https://phabricator.wikimedia.org/T359685#9625381 (10bd808) 05Open→03In progress a:03bd808 [01:13:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:45:28] (PuppetAgentNoResources) firing: No Puppet resources found on instance paws-puppetmaster-2 on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [01:55:35] (03CR) 10Tim Starling: "I am here because the modelines failed with an error in vim 8.2. I am not some sort of human modeline validator." [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1010664 (owner: 10Tim Starling) [02:06:28] (PuppetAgentFailure) firing: Puppet agent failure detected on instance toolsbeta-puppetdb-02 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [02:11:23] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review: [jobs-api,jobs-cli] Support job health checks - https://phabricator.wikimedia.org/T335592#9625402 (10CodeReviewBot) raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/16 Draft: [jobs-cli] support job healt... [02:12:02] (03PS2) 10Tim Starling: Add procps to base images [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1010690 [02:13:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:19:17] 10Wikibugs: Investigate producing a code quality report for GitLab based on flake8 - https://phabricator.wikimedia.org/T359685#9625413 (10bd808) With `--tee --output-file=dist/codeclimate.json --format=codeclimate` I can get it to write a codeclimate format json blob to the console, but for some reason writing t... [02:23:39] 10Wikibugs: Investigate producing a code quality report for GitLab based on flake8 - https://phabricator.wikimedia.org/T359685#9625426 (10bd808) It would really be ideal to have a formatter plugin that writes normal lines to screen and the codeclimate json to the output file. [02:23:41] (CloudVPSDesignateLeaks) resolved: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:26:49] (03PS3) 10Tim Starling: Fix broken vim modelines [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1010664 [02:28:04] 10Wikibugs: Investigate producing a code quality report for GitLab based on flake8 - https://phabricator.wikimedia.org/T359685#9625429 (10bd808) 05In progress→03Open a:05bd808→03None Unlicking and pushing this back to the backlog for now. Implementation is going to need some work upstream or finding a di... [02:28:42] 10Wikibugs: Convert `wikibugs2 phorge` to asyncio - https://phabricator.wikimedia.org/T359883#9625434 (10bd808) 05Open→03In progress a:03bd808 [03:51:28] (PuppetAgentFailure) resolved: Puppet agent failure detected on instance toolsbeta-puppetdb-02 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [04:00:28] (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 1 deleted instances on metricsinfra-puppetmaster-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [04:08:45] (03CR) 10Tim Starling: "I updated the commit message to clarify that this is a real error, not a quibble about the spec." [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1010664 (owner: 10Tim Starling) [04:45:28] (PuppetAgentNoResources) firing: No Puppet resources found on instance paws-puppetmaster-2 on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [05:06:28] (PuppetAgentFailure) firing: Puppet agent failure detected on instance toolsbeta-puppetdb-02 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [05:09:09] 05Grid-Engine-to-K8s-Migration: Migrate srwiki from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320057#9625567 (10dungodung) It didn't get executed for some reason. Or it did, but it encountered an error... [06:13:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:21:28] (PuppetAgentFailure) resolved: Puppet agent failure detected on instance toolsbeta-puppetdb-02 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [06:53:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:00:28] (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 1 deleted instances on metricsinfra-puppetmaster-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [07:13:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:23:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:45:28] (PuppetAgentNoResources) firing: No Puppet resources found on instance paws-puppetmaster-2 on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:54:28] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] build: Upgrade mediawiki/mediawiki-codesniffer to v43.0.0 [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1010662 (owner: 10Umherirrender) [08:13:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:23:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:14:55] 10Toolforge: Rust (buildservice) requires PHP - https://phabricator.wikimedia.org/T359937#9625757 (10Magnus) I created a custom `php.ini` and use it in the command: ` php8.1 -c /data/project/mix-n-match/mixnmatch_rs/php.ini /data/project/mix-n-match/scripts/person_dates/update_person_dates.php 48922 ` The php.in... [09:15:06] 10wikitech.wikimedia.org, 10Gerrit, 06Release-Engineering-Team, 07SecTeam-Processed, 07Security: wikitech hook to disable Gerrit user uses partial matches to identify account - https://phabricator.wikimedia.org/T307558#9625736 (10hashar) 05Resolved→03Open @bd808 **thank you so much for the fix**! Th... [09:34:20] 10Toolforge: Rust (buildservice) requires PHP - https://phabricator.wikimedia.org/T359937#9625800 (10dcaro) > Thanks @taavi Is there a list of packages required for a toolforge-equivalent php setup? For this the recommended way is using a php-only codebase, the issue is that you need both rust and php at the sa... [09:35:56] 10Toolforge: Rust (buildservice) requires PHP - https://phabricator.wikimedia.org/T359937#9625802 (10Magnus) For my own testing, is there a way to get a shell into the docker container? I have seen the docs for the `webservice` one, but that doesn't seem to fly for non-webservice ones? [09:37:12] 10Toolforge: Rust (buildservice) requires PHP - https://phabricator.wikimedia.org/T359937#9625803 (10dcaro) So, from within the container: ` # Using podman, docker would be the same dcaro@urcuchillay$ podman run --rm -ti --entrypoint launcher tools-harbor.wmcloud.org/tool-mix-n-match/tool-mix-n-match:latest bash... [09:38:18] 10Toolforge: Rust (buildservice) requires PHP - https://phabricator.wikimedia.org/T359937#9625804 (10dcaro) > For my own testing, is there a way to get a shell into the docker container? I have seen the docs for the webservice one, but that doesn't seem to fly for non-webservice ones? From your laptop you can p... [09:38:44] 10Toolforge, 07Epic: [jobs-api,webservice] Run webservices via the jobs framework - https://phabricator.wikimedia.org/T348755#9625805 (10taavi) [09:38:50] 10Toolforge: Make `webservice shell` a standalone tool - https://phabricator.wikimedia.org/T311917#9625806 (10taavi) [10:00:29] (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 1 deleted instances on metricsinfra-puppetmaster-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [10:50:28] (PuppetAgentNoResources) firing: No Puppet resources found on instance paws-puppetmaster-2 on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [11:06:28] (PuppetAgentFailure) firing: Puppet agent failure detected on instance toolsbeta-puppetdb-02 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [11:21:28] (PuppetAgentFailure) resolved: Puppet agent failure detected on instance toolsbeta-puppetdb-02 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [11:21:51] 10Toolforge: [webservice,toolforge-cli] Make `webservice shell` a standalone tool - https://phabricator.wikimedia.org/T311917#9625811 (10dcaro) p:05Triage→03Medium [11:26:34] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review: [jobs-api,buildservice-api,envvars-api] Investigate ways to present our multiple Openapi definitions to a future consolidated CLI client - https://phabricator.wikimedia.org/T354745#9625884 (10Slst2020) @dcaro How should we deal with the `security` top-l... [11:26:42] 10Toolforge: Rust (buildservice) requires PHP - https://phabricator.wikimedia.org/T359937#9625885 (10Magnus) I have some updates. With `php.ini` as ` extension_dir="/layers/fagiani_apt/apt/usr/lib/php/20210902" extension=mysqlnd extension=mysqli ` it loads the mysql library. Still fails with some password error... [11:28:11] 10Toolforge: Rust (buildservice) requires PHP - https://phabricator.wikimedia.org/T359937#9625899 (10dcaro) >>! In T359937#9625885, @Magnus wrote: > I have some updates. With `php.ini` as > ` > extension_dir="/layers/fagiani_apt/apt/usr/lib/php/20210902" > extension=mysqlnd > extension=mysqli > ` > it loads the... [11:30:03] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review: [jobs-api,buildservice-api,envvars-api] Investigate ways to present our multiple Openapi definitions to a future consolidated CLI client - https://phabricator.wikimedia.org/T354745#9625901 (10dcaro) >>! In T354745#9625884, @Slst2020 wrote: > @dcaro How... [11:32:20] 10Toolforge: [jobs-api] evaluate crossplane for composite objects creation and maintenance - https://phabricator.wikimedia.org/T360016 (10dcaro) 03NEW [11:33:48] 10Toolforge: [jobs-api] evaluate crossplane for composite objects creation and maintenance - https://phabricator.wikimedia.org/T360016#9625944 (10dcaro) [11:34:04] 10Toolforge: [jobs-api,buildservice-api,envvars-api] evaluate crossplane for composite objects creation and maintenance - https://phabricator.wikimedia.org/T360016#9625945 (10dcaro) [11:34:36] 10Toolforge: [jobs-api,buildservice-api,envvars-api] evaluate crossplane for composite objects creation and maintenance - https://phabricator.wikimedia.org/T360016#9625958 (10dcaro) [11:34:44] 10Toolforge: [jobs-api,buildservice-api,envvars-api] evaluate crossplane for composite objects creation and maintenance - https://phabricator.wikimedia.org/T360016#9625959 (10dcaro) [11:36:21] 10Toolforge: [jobs-api,buildservice-api,envvars-api] evaluate crossplane for composite objects creation and maintenance - https://phabricator.wikimedia.org/T360016#9625970 (10dcaro) [11:37:25] 10Toolforge: 14Rust (buildservice) requires PHP - 14https://phabricator.wikimedia.org/T359937#9625997 (10Magnus) 05Open→03Resolved a:03Magnus 14Got it working. For future reference: - [[ https://github.com/magnusmanske/mixnmatch_rs/blob/main/Aptfile | Aptfile ]] (not sure if all of these are neede... [11:39:29] 10Toolforge: 14Rust (buildservice) requires PHP - 14https://phabricator.wikimedia.org/T359937#9626010 (10dcaro) 14Small note that in order to make your code more future-proof, you might want to replace the hardcoded paths `/data/project/mix-n-match` with the path from the envvar `$TOOL_DATA_DIR`, not that i... [11:40:26] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review: [jobs-api,buildservice-api,envvars-api] Investigate ways to present our multiple Openapi definitions to a future consolidated CLI client - https://phabricator.wikimedia.org/T354745#9626018 (10Slst2020) `operationId`s need to be unique. At what level do... [11:40:54] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review: [jobs-api,buildservice-api,envvars-api] Investigate ways to present our multiple Openapi definitions to a future consolidated CLI client - https://phabricator.wikimedia.org/T354745#9626026 (10dcaro) >>! In T354745#9626018, @Slst2020 wrote: > `operationI... [11:41:26] 10Toolforge: [jobs-api,infra] upgrade all the existing toolforge jobs to the latest job version - https://phabricator.wikimedia.org/T359649#9626034 (10aborrero) We did a bit of research today about the size/scope of the problem: `lang=shell-session aborrero@tools-sgebastion-11:~$ kubectl sudo get cronjobs --all... [11:43:19] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review: [jobs-api,buildservice-api,envvars-api] Investigate ways to present our multiple Openapi definitions to a future consolidated CLI client - https://phabricator.wikimedia.org/T354745#9626052 (10Slst2020) >>! In T354745#9626026, @dcaro wrote: >>>! In T3547... [11:48:41] 10Tool-Global-user-contributions, 06Stewards-and-global-tools, 10Temporary accounts, 10XTools, 07Design: [Design EPIC] Global User Contributions - https://phabricator.wikimedia.org/T349901#9626073 (10KColeman-WMF) >>! In T349901#9622854, @Johannnes89 wrote: > I just clicked through the design brief. It s... [11:51:11] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review: [jobs-api,buildservice-api,envvars-api] Investigate ways to present our multiple Openapi definitions to a future consolidated CLI client - https://phabricator.wikimedia.org/T354745#9626097 (10dcaro) >>! In T354745#9626052, @Slst2020 wrote: >>>! In T3547... [12:06:25] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge (Toolforge iteration 07), 05Goal, 13Patch-For-Review: [infra] Decommission the Grid Engine infrastructure - https://phabricator.wikimedia.org/T314664#9626211 (10taavi) [12:06:33] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge (Toolforge iteration 07), 05Goal, 13Patch-For-Review: [infra] Decommission the Grid Engine infrastructure - https://phabricator.wikimedia.org/T314664#9626213 (10taavi) [12:42:20] 10Toolforge (Toolforge iteration 07), 10Projects-Cleanup, 13Patch-For-Review: [infra] Archive grid engine related Gerrit repositories - https://phabricator.wikimedia.org/T359935#9626251 (10taavi) a:03taavi [12:43:00] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge (Toolforge iteration 07): [infra] Archive grid engine related infrastructure tools - https://phabricator.wikimedia.org/T359934#9626260 (10taavi) [12:44:04] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9626266 (10MBH) I have merged your patch and successfully built the image, what should we do now? I tried to load `jobs.yaml` and ` tools.wikisaurusbot@tools-sgebast... [12:44:29] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge (Toolforge iteration 07), 05Goal, 13Patch-For-Review: [infra] Decommission the Grid Engine infrastructure - https://phabricator.wikimedia.org/T314664#9626270 (10taavi) [12:45:09] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9626272 (10MBH) One of scripts fails with this error: ` Traceback (most recent call last): File "/data/project/wikisaurusbot/facenapalmscripts/sandbox.py", line 12... [12:45:17] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge (Toolforge iteration 07), 05Goal, 13Patch-For-Review: [infra] Decommission the Grid Engine infrastructure - https://phabricator.wikimedia.org/T314664#9626277 (10taavi) [12:45:42] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9626280 (10taavi) You need to remove `lang=bash export PYTHONPATH=/data/project/shared/pywikibot/stable:/data/project/shared/pywikibot/stable/scripts ` from `.bash_p... [12:46:22] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9626287 (10dcaro) >>! In T320164#9626272, @MBH wrote: > One of scripts fails with this error: > ` > Traceback (most recent call last): > File "/data/project/wikisa... [12:47:10] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge (Toolforge iteration 07), 05Goal, 13Patch-For-Review: [infra] Decommission the Grid Engine infrastructure - https://phabricator.wikimedia.org/T314664#9626292 (10taavi) [12:48:06] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Toolforge: Introduce grid-less bookworm based bastion hosts - https://phabricator.wikimedia.org/T314665#9626301 (10taavi) [12:48:22] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge (Toolforge iteration 07), 05Goal, 13Patch-For-Review: [infra] Decommission the Grid Engine infrastructure - https://phabricator.wikimedia.org/T314664#9626302 (10taavi) [12:48:30] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Toolforge: Introduce grid-less bookworm based bastion hosts - https://phabricator.wikimedia.org/T314665#9626304 (10taavi) [12:48:38] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge (Toolforge iteration 07), 05Goal, 13Patch-For-Review: [infra] Decommission the Grid Engine infrastructure - https://phabricator.wikimedia.org/T314664#9626305 (10taavi) [12:48:54] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS (Debian Buster Deprecation), 10Toolforge, 07Epic, 05Goal: Toolforge: migrate to Debian Bullseye or later - https://phabricator.wikimedia.org/T311897#9626308 (10taavi) [12:49:02] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Toolforge: Introduce grid-less bookworm based bastion hosts - https://phabricator.wikimedia.org/T314665#9626307 (10taavi) [12:49:18] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9626315 (10MBH) I followed both instructions and load jobs.yaml again. Let's see if bots will run... [12:50:38] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9626328 (10dcaro) >>! In T320164#9626315, @MBH wrote: > I followed both instructions and load jobs.yaml again. Let's see if bots will run... There's some issues: `... [12:50:47] 06cloud-services-team, 10Striker: Deploy Striker instance for toolsbeta - https://phabricator.wikimedia.org/T360025 (10taavi) 03NEW [12:51:03] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9626342 (10dcaro) https://github.com/wikisaurus/wikisaurusbot/pull/2 there you go [12:51:59] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9626345 (10MBH) Thanks. Could you did something to `.yaml` file on Toolforge updated automatically when GitHub file updated, or in process of building? [12:52:39] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9626349 (10dcaro) >>! In T320164#9626345, @MBH wrote: > Thanks. Could you did something to `.yaml` file on Toolforge updated automatically when GitHub file updated,... [12:53:11] 05Grid-Engine-to-K8s-Migration: Migrate wmf-sitematrix from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320180#9626356 (10dcaro) @Bmueller found https://github.com/osmlab/wmf-sitematrix, seems like it's the repo you are using, I'll look into that [12:53:47] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge (Toolforge iteration 07), 05Goal, 13Patch-For-Review: [infra] Decommission the Grid Engine infrastructure - https://phabricator.wikimedia.org/T314664#9626364 (10taavi) [12:54:19] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9626378 (10MBH) ` * Job 'sandbox' (cronjob) (emails: onfailure) had 3 events: -- Pod 'sandbox-1710333455-mfbtf'. Phase: 'pending'. Container state: 'terminated'. S... [12:55:59] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9626384 (10MBH) Thank you very much for revitalasing my web tools. Could you explain what you did? Are they still runned through mono? If yes, why they run on python image, on... [12:56:01] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9626385 (10dcaro) >>! In T320164#9626378, @MBH wrote: > ` > * Job 'sandbox' (cronjob) (emails: onfailure) had 3 events: > -- Pod 'sandbox-1710333455-mfbtf'. Phase:... [12:57:05] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9626386 (10dcaro) I see this: ` 2024-03-13T12:56:14+00:00 [sandbox-1710334571-2hnv5] Skipped '/workspace/user-config.py': owned by someone else. ` I think that's so... [13:00:17] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9626394 (10MBH) > you have to add mount: all and filelogs: true to the entry in jobs.yaml for the files to get created It will be easier to use .err files like earli... [13:00:29] (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 1 deleted instances on metricsinfra-puppetmaster-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [13:00:34] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge (Toolforge iteration 07), 05Goal, 13Patch-For-Review: [infra] Decommission the Grid Engine infrastructure - https://phabricator.wikimedia.org/T314664#9626396 (10taavi) [13:06:00] 10Toolforge: 14Rust (buildservice) requires PHP - 14https://phabricator.wikimedia.org/T359937#9626432 (10Magnus) 14@dcaro thanks, done [13:13:15] (03PS1) 10Majavah: Archive repository [cloud/toolforge/grid-webservices] - 10https://gerrit.wikimedia.org/r/1010890 (https://phabricator.wikimedia.org/T359935) [13:13:22] (03CR) 10CI reject: [V: 04-1] Archive repository [cloud/toolforge/grid-webservices] - 10https://gerrit.wikimedia.org/r/1010890 (https://phabricator.wikimedia.org/T359935) (owner: 10Majavah) [13:14:28] (03PS1) 10Majavah: Archive repository [software/tools-manifest] - 10https://gerrit.wikimedia.org/r/1010891 (https://phabricator.wikimedia.org/T359935) [13:14:36] (03CR) 10CI reject: [V: 04-1] Archive repository [software/tools-manifest] - 10https://gerrit.wikimedia.org/r/1010891 (https://phabricator.wikimedia.org/T359935) (owner: 10Majavah) [13:15:10] (03CR) 10Majavah: [V: 03+2 C: 03+2] Archive repository [software/tools-manifest] - 10https://gerrit.wikimedia.org/r/1010891 (https://phabricator.wikimedia.org/T359935) (owner: 10Majavah) [13:15:30] (03CR) 10CI reject: [V: 04-1] Archive repository [software/tools-manifest] - 10https://gerrit.wikimedia.org/r/1010891 (https://phabricator.wikimedia.org/T359935) (owner: 10Majavah) [13:15:42] (03CR) 10Majavah: [V: 03+2 C: 03+2] Archive repository [cloud/toolforge/grid-webservices] - 10https://gerrit.wikimedia.org/r/1010890 (https://phabricator.wikimedia.org/T359935) (owner: 10Majavah) [13:15:50] (03CR) 10CI reject: [V: 04-1] Archive repository [cloud/toolforge/grid-webservices] - 10https://gerrit.wikimedia.org/r/1010890 (https://phabricator.wikimedia.org/T359935) (owner: 10Majavah) [13:16:26] (03PS2) 10Majavah: Archive repository [cloud/toolforge/grid-webservices] - 10https://gerrit.wikimedia.org/r/1010890 (https://phabricator.wikimedia.org/T359935) [13:16:33] (03CR) 10CI reject: [V: 04-1] Archive repository [cloud/toolforge/grid-webservices] - 10https://gerrit.wikimedia.org/r/1010890 (https://phabricator.wikimedia.org/T359935) (owner: 10Majavah) [13:16:44] (03CR) 10Majavah: [V: 03+2 C: 03+2] Archive repository [cloud/toolforge/grid-webservices] - 10https://gerrit.wikimedia.org/r/1010890 (https://phabricator.wikimedia.org/T359935) (owner: 10Majavah) [13:16:51] (03CR) 10CI reject: [V: 04-1] Archive repository [cloud/toolforge/grid-webservices] - 10https://gerrit.wikimedia.org/r/1010890 (https://phabricator.wikimedia.org/T359935) (owner: 10Majavah) [13:17:59] 10Toolforge (Toolforge iteration 07), 10Projects-Cleanup, 13Patch-For-Review: 14[infra] Archive grid engine related Gerrit repositories - 14https://phabricator.wikimedia.org/T359935#9626487 (10taavi) [13:18:04] 10Toolforge (Toolforge iteration 07), 10Projects-Cleanup, 13Patch-For-Review: 14[infra] Archive grid engine related Gerrit repositories - 14https://phabricator.wikimedia.org/T359935#9626488 (10taavi) 05Open→03Resolved [13:18:11] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge (Toolforge iteration 07), 05Goal, 13Patch-For-Review: [infra] Decommission the Grid Engine infrastructure - https://phabricator.wikimedia.org/T314664#9626489 (10taavi) [13:35:26] (03PS2) 10Arturo Borrero Gonzalez: toolforge.k8s.prepare_upgrade: be more flexible when checking repository things [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1010558 (https://phabricator.wikimedia.org/T359927) [13:38:28] (03PS3) 10Arturo Borrero Gonzalez: toolforge.k8s.prepare_upgrade: be more flexible when checking repository things [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1010558 (https://phabricator.wikimedia.org/T359927) [13:39:09] (03PS4) 10Arturo Borrero Gonzalez: toolforge.k8s.prepare_upgrade: be more flexible when checking repository things [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1010558 (https://phabricator.wikimedia.org/T359927) [13:47:54] (03PS5) 10Arturo Borrero Gonzalez: toolforge.k8s.prepare_upgrade: be more flexible when checking repository things [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1010558 (https://phabricator.wikimedia.org/T359927) [13:53:12] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9626630 (10MBH) A `sandbox` job spams my mail every ~15 mins about its failures, it will be good to fix it. [13:53:39] 10Toolforge: [envvars-cli] Either hide or show envvars values, but not both - https://phabricator.wikimedia.org/T359558#9626635 (10EBomani) a:03EBomani [13:55:28] (PuppetAgentNoResources) firing: No Puppet resources found on instance paws-puppetmaster-2 on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [13:56:33] (03PS6) 10Arturo Borrero Gonzalez: toolforge.k8s.prepare_upgrade: be more flexible when checking repository things [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1010558 (https://phabricator.wikimedia.org/T359927) [13:57:44] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9626649 (10dcaro) Got a working patch https://github.com/wikisaurus/wikisaurusbot/pull/3 It was kind of tricky to get pywikibot auth working though. Let's see if th... [13:59:20] 05Grid-Engine-to-K8s-Migration, 10Wiki-Loves-Monuments-Database: Migrate heritage from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319787#9626657 (10Bmueller) @dcaro I believe it's this one: https://gerrit.wikimedia.org/r/q/project:labs%252Ftools%252Fheritage [14:02:02] (03CR) 10Majavah: [C: 03+1] "see inline" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1010558 (https://phabricator.wikimedia.org/T359927) (owner: 10Arturo Borrero Gonzalez) [14:03:23] (03PS7) 10Arturo Borrero Gonzalez: toolforge.k8s.prepare_upgrade: be more flexible when checking repository things [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1010558 (https://phabricator.wikimedia.org/T359927) [14:05:42] (03CR) 10Arturo Borrero Gonzalez: toolforge.k8s.prepare_upgrade: be more flexible when checking repository things (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1010558 (https://phabricator.wikimedia.org/T359927) (owner: 10Arturo Borrero Gonzalez) [14:11:07] (03CR) 10Majavah: [C: 03+1] toolforge.k8s.prepare_upgrade: be more flexible when checking repository things [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1010558 (https://phabricator.wikimedia.org/T359927) (owner: 10Arturo Borrero Gonzalez) [14:13:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:17:45] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9626782 (10dcaro) >>! In T319883#9626384, @MBH wrote: > Thank you very much for revitalasing my web tools. Could you explain what you did? Are they still runned through mono?... [14:33:36] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9626820 (10MBH) I tried to rewrite my tools using standard dotnet way to create web tools, but stuck on first steps. But why it's impossible to run this tools on k8s exactly... [14:36:28] (PuppetAgentFailure) firing: Puppet agent failure detected on instance toolsbeta-puppetdb-02 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [14:39:47] siddharthvp closed https://github.com/toolforge/quarry/pull/32 [14:40:01] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9626834 (10MBH) ` Sleeping for 8.4 seconds, 2024-03-13 14:25:06 Page [[Обсуждение Википедии:Песочница]] saved Sleeping for 9.5 seconds, 2024-03-13 14:25:15 Page [[Ин... [14:43:37] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9626851 (10dcaro) >>! In T319883#9626820, @MBH wrote: > I tried to rewrite my tools using standard dotnet way to create web tools, but stuck on first steps. > > But why it's... [14:45:23] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9626857 (10dcaro) >>! In T320164#9626834, @MBH wrote: > ` > Sleeping for 8.4 seconds, 2024-03-13 14:25:06 > Page [[Обсуждение Википедии:Песочница]] saved > Sleeping... [14:47:58] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9626859 (10dcaro) As in, can you elaborate on what did not work? [14:52:06] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9626874 (10MBH) Current error: ` Traceback (most recent call last): File "/workspace/facenapalmscripts/sandbox.py", line 49, in main() File "/worksp... [14:58:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:59:43] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9626898 (10dcaro) >>! In T320164#9626874, @MBH wrote: > Current error: > ` > Traceback (most recent call last): > File "/workspace/facenapalmscripts/sandbox.py", l... [15:02:40] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9626902 (10dcaro) I think this might fix it: https://github.com/wikisaurus/wikisaurusbot/pull/4 [15:03:38] 10Wikibugs: AttributeError: 'GerritMessageBuilder' object has no attribute 'esc' - https://phabricator.wikimedia.org/T360038 (10taavi) 03NEW [15:05:54] 06cloud-services-team, 10Toolforge (Toolforge iteration 07), 13Patch-For-Review: 14[cookbook] wmcs.toolforge.k8s.prepare_upgrade: be more flexible checking for deb package components - 14https://phabricator.wikimedia.org/T359927#9626917 (10aborrero) 05Open→03Resolved [15:05:59] 06cloud-services-team, 10Toolforge (Toolforge iteration 07): Upgrade Toolforge Kubernetes to version 1.24 - https://phabricator.wikimedia.org/T307651#9626918 (10aborrero) [15:13:11] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:20:39] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9626941 (10MBH) My building process was just clicking on green arrow on Visual Studio panel, my "the tools installed that you used during the build" was just plain free (as in... [15:23:11] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:31:46] 05Grid-Engine-to-K8s-Migration, 06Growth-Team, 10Community-Tech (CommTech-Kanban): 14Migrate ERANBOT project off of Grid Engine - 14https://phabricator.wikimedia.org/T306888#9626969 (10taavi) 14>>! In T306888#9623696, @TheresNoTime wrote: > b) that nothing remains on the grid engine :-) As far as I can... [15:33:32] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9626971 (10MBH) Merged, rebuilded, loaded. Current errors in two tools: ` NameError: name 'latest_revision' is not defined CRITICAL: Exiting due to uncaught exceptio... [15:36:16] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9626976 (10dcaro) >>! In T320164#9626971, @MBH wrote: > Merged, rebuilded, loaded. Current errors in two tools: > ` > NameError: name 'latest_revision' is not define... [15:43:49] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9627011 (10dcaro) https://github.com/wikisaurus/wikisaurusbot/pull/5 [15:48:02] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.23 to 1.24 (T359638) [15:48:07] T359638: [toolsbeta,infra] upgrade kubernetes to 1.24 - https://phabricator.wikimedia.org/T359638 [15:49:42] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.23 to 1.24 (T359638) [15:58:13] 10Quarry: Set query result retention time - https://phabricator.wikimedia.org/T360041 (10rook) 03NEW [16:00:08] 10Wikibugs: AttributeError: 'GerritMessageBuilder' object has no attribute 'esc' - https://phabricator.wikimedia.org/T360038#9627040 (10bd808) 05Open→03In progress p:05Triage→03High a:03bd808 [16:00:29] (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 1 deleted instances on metricsinfra-puppetmaster-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [16:02:09] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9627047 (10dcaro) >>! In T319883#9626941, @MBH wrote: > My building process was just clicking on green arrow on Visual Studio panel, my "the tools installed that you used duri... [16:06:22] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9627060 (10MBH) One ruwiki user on our discord server noticed this change on your first PR, it looks like syntax error: {F42599930} This file: https://github.com/wik... [16:06:42] 10Wikibugs, 13Patch-For-Review: AttributeError: 'GerritMessageBuilder' object has no attribute 'esc' - https://phabricator.wikimedia.org/T360038#9627057 (10CodeReviewBot) bd808 opened https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/18 GerritMessageBuilder: fix `self` reference missed i... [16:06:49] 10Wikibugs, 13Patch-For-Review: AttributeError: 'GerritMessageBuilder' object has no attribute 'esc' - https://phabricator.wikimedia.org/T360038#9627066 (10CodeReviewBot) bd808 merged https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/18 GerritMessageBuilder: fix `self` reference missed i... [16:10:16] 10Wikibugs, 13Patch-For-Review: AttributeError: 'GerritMessageBuilder' object has no attribute 'esc' - https://phabricator.wikimedia.org/T360038#9627092 (10bd808) Slow down and read the code Bryan. :(( ` [pod/gerrit-6579d9d999-g7bl7/job] Traceback (most recent call last): [pod/gerrit-6579d9d999-g7bl7/job] Fi... [16:14:46] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 (T359638) [16:14:51] T359638: [toolsbeta,infra] upgrade kubernetes to 1.24 - https://phabricator.wikimedia.org/T359638 [16:14:52] !log aborrero@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 (T359638) [16:15:17] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 (T359638) [16:15:50] !log aborrero@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 (T359638) [16:16:02] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9627125 (10dcaro) Done https://github.com/wikisaurus/wikisaurusbot/pull/6, did the previous changes work though? [16:21:02] (03CR) 10Arturo Borrero Gonzalez: [C:03+1] "LGTM." [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1010901 (owner: 10Majavah) [16:26:46] (03CR) 10Majavah: [C:03+2] kubernetes: Validate K8s versions are valid [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1010901 (owner: 10Majavah) [16:37:17] (03Merged) 10jenkins-bot: kubernetes: Validate K8s versions are valid [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1010901 (owner: 10Majavah) [16:42:55] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9627249 (10MBH) If you could rewrite any one of my tools to follow modern dotnet way to build web apps, it will be very helpful to me and I think I will rewrite all other tool... [16:46:59] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9627256 (10dcaro) >>! In T319883#9627249, @MBH wrote: > If you could rewrite any one of my tools to follow modern dotnet way to build web apps, it will be very helpful to me a... [16:52:48] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9627283 (10MBH) I started to trying to rewrite my tools on January and did some work, looks like new code should be something like this: ` var builder = WebApplication.CreateB... [16:55:28] (PuppetAgentNoResources) firing: No Puppet resources found on instance paws-puppetmaster-2 on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [16:56:52] (03PS1) 10Arturo Borrero Gonzalez: wmcs_libs/k8s/kubernetes: fix namespace flags cannot be placed before plugin [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1010914 [17:03:22] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9627327 (10MBH) If `Skipped '/workspace/user-config.py': owned by someone else.` isn't an issue, I don't see problems now on three bots. Bot edits started: https://... [17:05:25] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9627340 (10MBH) But one bot still has this error: ` Traceback (most recent call last): File "/workspace/facenapalmscripts/autopurge.py", line 83, in m... [17:11:01] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9627352 (10dcaro) >>! In T320164#9627340, @MBH wrote: > But one bot still has this error: > ` > Traceback (most recent call last): > File "/workspace/facenapalmscr... [17:14:06] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9627393 (10MBH) A file `autopurge-hourly.err` contains this error many times (since we revitalised this bots), so I assume it can be related to bot code. Maybe some... [17:20:40] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9627414 (10dcaro) >>! In T320164#9627404, @dcaro wrote: >>>! In T320164#9627393, @MBH wrote: >> A file `autopurge-hourly.err` contains this error many times (since w... [17:36:03] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9627540 (10dcaro) Do you know if it's actually doing things? (I have no idea of Russian, so I'm having a hard time navigating the wiki xd) [17:38:51] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9627570 (10MBH) It (English) name means it purges pages, i.e. did null edits, so its edits will not be seen in its contribs. Its code written on Python, not Russian,... [17:39:28] (PuppetAgentNoResources) firing: No Puppet resources found on instance tools-sgeexec-10-22 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [17:44:28] (PuppetAgentNoResources) firing: (8) No Puppet resources found on instance tools-db-3 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [17:45:41] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9627629 (10dcaro) >>! In T320164#9627570, @MBH wrote: > It (English) name means it purges pages, i.e. did null edits, so its edits will not be seen in its contribs.... [17:46:47] (03CR) 10Majavah: [C:03+1] wmcs_libs/k8s/kubernetes: fix namespace flags cannot be placed before plugin [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1010914 (owner: 10Arturo Borrero Gonzalez) [17:49:28] (PuppetAgentNoResources) firing: (10) No Puppet resources found on instance tools-db-3 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [17:50:36] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9627636 (10MBH) It, obviously, https://ru.wikipedia.org/wiki/К:Википедия:Страницы_с_ежечасно_очищаемым_кэшем and https://ru.wikipedia.org/wiki/Шаблон:Очищать_кэш/статус [17:53:21] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9627639 (10MBH) I don't know how to find out if it purged any pages. [17:54:10] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9627642 (10dcaro) >>! In T320164#9627636, @MBH wrote: > It, obviously, https://ru.wikipedia.org/wiki/К:Википедия:Страницы_с_ежечасно_очищаемым_кэшем and https://ru.w... [17:54:28] (PuppetAgentNoResources) firing: (12) No Puppet resources found on instance tools-db-3 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [17:55:46] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9627649 (10dcaro) Oh, it's running with `--nolog` that avoids logging there actually :/, so I don't know then how to find out either [17:55:58] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9627650 (10dcaro) I can try modifying it a bit tomorrow to have some logs see where it's getting stuck if it is. [17:59:28] (PuppetAgentNoResources) firing: (31) No Puppet resources found on instance tools-acme-chief-01 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [18:00:56] 05Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9627662 (10MBH) You could open any page you have questions about just by entering its title into search field on ruwiki and clicking Enter. "К" is a ruwiki's alias f... [18:02:29] 10Wikibugs, 13Patch-For-Review: AttributeError: 'GerritMessageBuilder' object has no attribute 'esc' - https://phabricator.wikimedia.org/T360038#9627663 (10CodeReviewBot) bd808 opened https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/19 GerritMessageBuilder: add much needed output tests [18:04:51] 10Wikibugs, 13Patch-For-Review: AttributeError: 'GerritMessageBuilder' object has no attribute 'esc' - https://phabricator.wikimedia.org/T360038#9627670 (10CodeReviewBot) bd808 merged https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/19 GerritMessageBuilder: add much needed output tests [18:08:19] 10ToolforgeBundle, 10SVG Translate Tool, 10Community-Tech (CommTech-Kanban): 14Add Toolforge-specific deploy script for running npm in a separate job - 14https://phabricator.wikimedia.org/T346009#9627678 (10JWheeler-WMF) 05Open→03Resolved [18:09:28] (PuppetAgentNoResources) firing: (37) No Puppet resources found on instance tools-acme-chief-01 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [18:14:28] (PuppetAgentNoResources) firing: (38) No Puppet resources found on instance tools-acme-chief-01 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [18:19:28] (PuppetAgentNoResources) firing: (43) No Puppet resources found on instance tools-acme-chief-01 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [18:24:28] (PuppetAgentNoResources) firing: (42) No Puppet resources found on instance tools-acme-chief-01 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [18:29:28] (PuppetAgentNoResources) firing: (32) No Puppet resources found on instance tools-acme-chief-01 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [18:34:28] (PuppetAgentNoResources) resolved: (21) No Puppet resources found on instance tools-k8s-etcd-18 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [18:53:35] 10Quarry, 10AutoWikiBrowser: Quarry run result in AWB make list - https://phabricator.wikimedia.org/T134141#9627803 (10Reedy) [19:00:29] (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 1 deleted instances on metricsinfra-puppetmaster-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [19:17:50] (ProbeDown) firing: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:22:50] (ProbeDown) resolved: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:32:47] 10Wikibugs: 14AttributeError: 'GerritMessageBuilder' object has no attribute 'esc' - 14https://phabricator.wikimedia.org/T360038#9628334 (10bd808) 05In progress→03Resolved [19:53:01] 06cloud-services-team, 10VPS-Projects, 10Puppet (Puppet 7.0): Migrate Puppet servers in Cloud Services team managed projects to Puppet 7 - https://phabricator.wikimedia.org/T351453#9628364 (10Andrew) [20:00:28] (PuppetAgentNoResources) firing: No Puppet resources found on instance paws-puppetmaster-2 on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [20:56:05] 10Wikibugs: Explore replacing asyncio-redis with redis.asyncio - https://phabricator.wikimedia.org/T360074 (10bd808) 03NEW p:05Triage→03Medium [21:59:28] (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance bastion in project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [22:00:29] (PuppetStaleCertificates) firing: Found non-revoked Puppet certificates for 1 deleted instances on metricsinfra-puppetmaster-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [22:04:28] (PuppetAgentStaleLastRun) firing: (2) Last Puppet run was over 24 hours ago on instance bastion in project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [22:41:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:46:41] (CloudVPSDesignateLeaks) firing: (5) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:50:50] (ProbeDown) firing: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:51:41] (CloudVPSDesignateLeaks) firing: (5) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:55:50] (ProbeDown) resolved: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:56:41] (CloudVPSDesignateLeaks) resolved: (5) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:00:28] (PuppetAgentNoResources) firing: No Puppet resources found on instance paws-puppetmaster-2 on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources