[00:19:01] (03open) 10samwilson: Switch CI from GitHub to GitLab [toolforge-repos/wabmd] - 10https://gitlab.wikimedia.org/toolforge-repos/wabmd/-/merge_requests/1 [00:21:00] (03merge) 10samwilson: Switch CI from GitHub to GitLab [toolforge-repos/wabmd] - 10https://gitlab.wikimedia.org/toolforge-repos/wabmd/-/merge_requests/1 [01:24:34] FIRING: [3x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-2 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesse [01:32:30] 06cloud-services-team, 10Data-Services: Wikidata's term tables missing in Quarry - https://phabricator.wikimedia.org/T401049#11055801 (10JJMC89) 05Open→03Invalid These tables are in a separate database. See [[ https://wikitech.wikimedia.org/wiki/News/2025_Wikidata_term_store_database_split. | 2025 Wiki... [04:02:13] 06cloud-services-team, 10Toolforge: Investigate daily disconnections of IRC bots hosted in Toolforge - https://phabricator.wikimedia.org/T400223#11055809 (10Danilo) I made another test now, I restarted wmopbot one minute before the time it is disconnected every day, to see if the problem only affect long lasti... [06:31:30] FIRING: NfsAlmostFull: The NFS drive is over 85% capacity (currently 87.4%) at host paws-nfs-1 in project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DNfsAlmostFull [08:51:57] 06cloud-services-team, 10Data-Services: Wikidata's term tables missing in Quarry - https://phabricator.wikimedia.org/T401049#11055855 (10Bdijkstra) Okay thanks. Via the talk page there I found out that there's a task for Quarry to get updated: [[ https://phabricator.wikimedia.org/T395800 | T395800 ]]. Sho... [10:07:30] 10Tool-archive-externa-links, 10Wikidata, 10Wikidata-Gadgets: [Code] Script utilisateur ArchiveExternaLinks avec Internet Archive - https://phabricator.wikimedia.org/T399885#11055899 (10paulwiki) [10:08:58] 10Tool-archive-externa-links, 10Wikidata, 10Wikidata-Gadgets: [Code] Script utilisateur ArchiveExternaLinks - https://phabricator.wikimedia.org/T399885#11055900 (10paulwiki) [11:08:02] 06Toolforge-standards-committee: Adoption request for pagelister - https://phabricator.wikimedia.org/T398111#11055924 (10LucasWerkmeister) @MusikAnimal per the above, you should be able to `become toolforge-standards-committee become pagelister` now. I think it’s also fine for me to confirm publicly that the `.... [12:44:57] 06cloud-services-team, 10Toolforge, 06Toolforge-standards-committee: Keep track of tools without stated default licenses - https://phabricator.wikimedia.org/T190377#11055958 (10Pintoch) The [[ https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Tool_sweep | Tool sweep ]] is an initiative initiated by @Legok... [14:31:30] FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on toolsbeta-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [14:44:21] (03open) 10don-vip: Draft: Detect GPS EXIF data and set {{GPS EXIF}} [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/3 [14:58:12] (03update) 10don-vip: Draft: Detect GPS EXIF data and set {{GPS EXIF}} [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/3 [14:58:34] (03update) 10don-vip: Draft: Detect GPS EXIF data; set {{GPS EXIF}} and {{Location}} [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/3 [15:21:52] FIRING: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:26:52] RESOLVED: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:52:18] (03update) 10don-vip: Draft: Detect GPS EXIF data; set {{GPS EXIF}} and {{Location}} [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/3 [15:52:44] (03update) 10don-vip: Detect GPS EXIF data; set {{GPS EXIF}} and {{Location}} [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/3 [16:00:21] (03merge) 10don-vip: Detect GPS EXIF data; set {{GPS EXIF}} and {{Location}} [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/3 [16:17:12] 06cloud-services-team, 10Toolforge: Ease `toolforge_weld` usage from within pods - https://phabricator.wikimedia.org/T401065#11056135 (10DamianZaremba) [16:17:50] 06cloud-services-team, 10Toolforge: Ease `toolforge_weld` usage from within pods - https://phabricator.wikimedia.org/T401065#11056137 (10DamianZaremba) Sorry, I forgot that it doesn't pick up the context of the workboard currently being viewed... added Toolforge. [17:19:31] 06cloud-services-team, 10Toolforge: Toolforge jobs api reports status as running before container is launched - https://phabricator.wikimedia.org/T401069 (10DamianZaremba) 03NEW [18:00:43] 06cloud-services-team, 10Toolforge: Allow health checks to be specified for one-off jobs - https://phabricator.wikimedia.org/T401071 (10DamianZaremba) 03NEW [18:12:01] (03open) 10bodhisattwa: add de from Lydia [toolforge-repos/sangkalak] - 10https://gitlab.wikimedia.org/toolforge-repos/sangkalak/-/merge_requests/4 [19:18:39] 06cloud-services-team, 10Toolforge: `toolforge jobs logs` returns nothing if started too early. - https://phabricator.wikimedia.org/T401073#11056292 (10DamianZaremba) [19:21:06] 06cloud-services-team, 10Toolforge: [toolforge,jobs] "toolforge jobs logs" fails when job has not started yet - https://phabricator.wikimedia.org/T349775#11056296 (10DamianZaremba) This is not the same error and the backend is likely different (loki vs kubernetes due to no `--follow`), but T401073 is sort of r... [19:28:54] 06cloud-services-team, 10Toolforge: Support installing packages from non-upstream repo and/or build pack for C/C++code - https://phabricator.wikimedia.org/T401075 (10DamianZaremba) 03NEW [19:31:21] 06cloud-services-team, 10Toolforge: Allow health checks (readinessProbe) to be specified for one-off jobs - https://phabricator.wikimedia.org/T401071#11056316 (10DamianZaremba) [19:53:28] 06cloud-services-team, 10Toolforge: [toolforge,storage,swift,s3] Object store? - https://phabricator.wikimedia.org/T225190#11056319 (10DamianZaremba) As a concrete example `cluebotng-trainer` currently uses NFS for storing artefacts from training runs; this would easy fit into an object store model allowing 0... [20:41:11] 10PAWS: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T401076#11056335 (10LibUp-bot) [20:41:19] 06cloud-services-team, 10Toolforge: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T401077#11056337 (10LibUp-bot) [23:14:23] 10Toolforge (Toolforge iteration 22): Allow querying job logs for completed jobs - https://phabricator.wikimedia.org/T401079 (10DamianZaremba) 03NEW [23:15:02] 06cloud-services-team, 10Toolforge: Allow querying job logs for completed jobs - https://phabricator.wikimedia.org/T401079#11056407 (10DamianZaremba) [23:16:01] 06cloud-services-team, 10Toolforge: Allow querying job logs for completed jobs - https://phabricator.wikimedia.org/T401079#11056410 (10DamianZaremba) This is perhaps covered by "non-existant" in T400913 [23:34:42] 06cloud-services-team, 10Toolforge: Allow querying job logs for completed jobs - https://phabricator.wikimedia.org/T401079#11056446 (10JJMC89) →14Duplicate dup:03T400913 [23:34:46] 06cloud-services-team, 10Toolforge (Toolforge iteration 22), 13Patch-For-Review: [jobs-api] Allow querying logs for non-existent jobs - https://phabricator.wikimedia.org/T400913#11056448 (10JJMC89)