[00:00:32] <wikibugs>	 10Tool-wikinewsie: Regional and thematic portals for Wikinewsie - https://phabricator.wikimedia.org/T432162#12193785 (10Habst) Done, for example at https://wikinewsie.toolforge.org/en/portal/sports
[00:00:48] <wikibugs>	 10Tool-wikinewsie: Regional and thematic portals for Wikinewsie - https://phabricator.wikimedia.org/T432162#12193786 (10Habst) 05In progress→03Resolved
[00:18:53] <wikibugs>	 (03update) 10don-vip: Add NASA GAPE [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/33
[00:19:02] <wikibugs>	 (03merge) 10don-vip: Add NASA GAPE [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/33
[02:24:16] <wikibugs>	 10Tools: Consultations Stats tool is not finding everyone (different, smaller number of participants each time) - https://phabricator.wikimedia.org/T433785#12193863 (10WhatamIdoing) Thank you, @samuelguebo.   It's a bit slower (less than a minute) but it gets the right answer.  I don't need it to be very quick....
[02:24:38] <wikibugs>	 10Tools: Consultations Stats tool is not finding everyone (different, smaller number of participants each time) - https://phabricator.wikimedia.org/T433785#12193864 (10WhatamIdoing) 05In progress→03Resolved
[05:01:43] <wikibugs>	 10Tool-campwiz-nxt: CampWiz Nxt Redesign: Root Path - https://phabricator.wikimedia.org/T415408#12193959 (10MohammadAlawneh) Hi ,  Thank you very much for your kind reply and for taking the time to answer my question.  I really appreciate the helpful resources you shared. I understand that the GSoC 2026 appl...
[06:03:33] <wikibugs>	 06tools-infrastructure-team: Review capacity projection setup for toolforge and cloud-vps - https://phabricator.wikimedia.org/T434128#12193999 (10BLiviero-WMF) initial thoughts... - I am not sure how to internalize "subscription" (or over-subscription) and the risks associated with it - it would be helpful to ge...
[06:19:24] <wikibugs>	 (03merge) 10earwig: Use Toolforge CDNjs mirror for loading jQuery [toolforge-repos/sigma] - 10https://gitlab.wikimedia.org/toolforge-repos/sigma/-/merge_requests/1 (owner: 10taavi)
[08:41:43] <wikibugs>	 10Tool-anagrimes, 06All-and-every-Wiktionary, 10MediaWiki-extension-requests: Create a MediaWiki extension to supersede https://anagrimes.toolforge.org/hasard.php - https://phabricator.wikimedia.org/T154730#12194315 (10Aklapper) p:05Medium→03Low
[13:01:30] <wikibugs>	 10tool-wdlocator: No data from wikidata is loaded - https://phabricator.wikimedia.org/T426903#12195150 (10Strubbl) I fear this issue is active. Maybe because of the [[ https://community.openstreetmap.org/t/overpass-api-performance-issues/140598 | overload of the overpass API ]]?
[13:08:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[14:18:37] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops: cloudvirt1071 crash, again - https://phabricator.wikimedia.org/T431374#12195405 (10Andrew) p:05Triage→03Medium
[14:18:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[14:29:34] <wmcs-alerts>	 RESOLVED: ToolsNfsAlmostFull: Toolforge NFS is almost full - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsNfsAlmostFull  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsNfsAlmostFull
[14:38:01] <wikibugs>	 (03open) 10andrew: Add v6 PTR for mail.wikimedia.az [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/343 (https://phabricator.wikimedia.org/T432761)
[14:38:30] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Add IPv6 PTR record for mail.wikimedia.az - https://phabricator.wikimedia.org/T432761#12195472 (10Andrew) a:03Andrew
[14:41:43] <wikibugs>	 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Network unavailable on a few VMs - https://phabricator.wikimedia.org/T432426#12195493 (10Andrew) Bookworm example today: demo-patrollers.wmf-research-tools.eqiad1.wikimedia.cloud
[14:42:35] <wikibugs>	 (03update) 10andrew: Add v6 PTR for mail.wikimedia.az [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/343 (https://phabricator.wikimedia.org/T432761)
[15:25:10] <wikibugs>	 10Tool-wiki-gender-stats: Change link to contribution guide - https://phabricator.wikimedia.org/T432532#12195634 (10Wyslijp16) I guess I did [[ https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/commit/b6ddce671500c432c6fef5116e05ccfd63c6097e | something ]].
[15:43:14] <wikibugs>	 10Tool-wiki-gender-stats: Add alternative texts to images on the website - https://phabricator.wikimedia.org/T434311 (10Wyslijp16) 03NEW
[15:43:20] <wikibugs>	 10Tool-wiki-gender-stats: Add alternative texts to images on the website - https://phabricator.wikimedia.org/T434311#12195715 (10Wyslijp16) p:05Triage→03Lowest
[16:12:24] <wikibugs>	 10Tool-wiki-gender-stats: Make chart labels filtering more intuitive - https://phabricator.wikimedia.org/T434318 (10Danya) 03NEW
[16:12:35] <wikibugs>	 10Tool-wiki-gender-stats: Make chart labels filtering more intuitive - https://phabricator.wikimedia.org/T434318#12195897 (10Danya)
[16:12:44] <wikibugs>	 10Tool-wiki-gender-stats: Make chart labels filtering more intuitive - https://phabricator.wikimedia.org/T434318#12195899 (10Danya) p:05Triage→03Medium
[16:41:51] <wikibugs>	 (03update) 10danyya: Draft: Resolve T426057 "Translate language names in genders-by-wiki page" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/97
[17:18:30] <wikibugs>	 10Tool-wiki-gender-stats: Change link to contribution guide - https://phabricator.wikimedia.org/T432532#12196103 (10Danya)
[17:18:31] <wikibugs>	 10Tool-wiki-gender-stats: [VERSION] 0.5.0 - https://phabricator.wikimedia.org/T432922#12196104 (10Danya)
[17:18:32] <wikibugs>	 10Tool-wiki-gender-stats: [VERSION] 0.4.0 - https://phabricator.wikimedia.org/T432521#12196105 (10Danya)
[17:19:02] <wikibugs>	 10Tool-wiki-gender-stats: Change link to contribution guide - https://phabricator.wikimedia.org/T432532#12196106 (10Danya) a:03Wyslijp16
[17:20:05] <wikibugs>	 10Tool-wiki-gender-stats: Change link to contribution guide - https://phabricator.wikimedia.org/T432532#12196107 (10Danya) Please @Wyslijp16, could you next time follow the [[ https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/blob/develop/CONTRIBUTING.md#contribution-workflow | proper procedure ]]?
[17:22:37] <wikibugs>	 10Tool-wiki-gender-stats: Change link to contribution guide - https://phabricator.wikimedia.org/T432532#12196126 (10Danya) 05Open→03Resolved
[17:24:50] <wikibugs>	 (03merge) 10andrew: Handle tools where home directory was archived without DB updates [repos/cloud/toolforge/disable-tool] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/disable-tool/-/merge_requests/27 (https://phabricator.wikimedia.org/T431241) (owner: 10taavi)
[17:47:13] <wikibugs>	 10Tool-wiki-gender-stats: Add alternative texts to images on the website - https://phabricator.wikimedia.org/T434311#12196205 (10Danya) 05Open→03Resolved a:03Danya It’s a bit too late now, this should have been done earlier. Closing.
[17:47:52] <wikibugs>	 10Tool-wiki-gender-stats: Add alternative texts to images on the website - https://phabricator.wikimedia.org/T434311#12196209 (10Danya) 05Resolved→03Open Woops I misread. Re-opening.
[17:48:22] <wikibugs>	 10Tool-wiki-gender-stats: Add alternative texts to images on the website - https://phabricator.wikimedia.org/T434311#12196212 (10Danya)
[17:48:52] <wikibugs>	 10Tool-wiki-gender-stats: Add alternative texts to images on the website - https://phabricator.wikimedia.org/T434311#12196213 (10Danya)
[17:49:04] <wikibugs>	 10Tool-wiki-gender-stats: Add alternative texts to images on the website - https://phabricator.wikimedia.org/T434311#12196214 (10Danya) a:05Danya→03None
[17:51:57] <wikibugs>	 (03update) 10danyya: Draft: Resolve T426057 "Translate language names in genders-by-wiki page" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/97
[17:53:04] <wikibugs>	 (03update) 10danyya: Resolve T426057 "Translate language names in genders-by-wiki page" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/97
[17:53:13] <wikibugs>	 (03merge) 10danyya: Resolve T426057 "Translate language names in genders-by-wiki page" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/97
[17:53:38] <wikibugs>	 10Tool-wiki-gender-stats: Change link to contribution guide - https://phabricator.wikimedia.org/T432532#12196223 (10Wyslijp16) Sorry, I will do it next time.
[17:53:40] <wikibugs>	 10Tool-wiki-gender-stats, 13Patch-For-Review: Translate language names in genders-by-wiki page - https://phabricator.wikimedia.org/T426057#12196220 (10Danya) 05In progress→03Resolved
[17:54:44] <wikibugs>	 10Tool-wiki-gender-stats: Change link to contribution guide - https://phabricator.wikimedia.org/T432532#12196224 (10Danya) Thank you :)
[18:02:25] <wikibugs>	 10Tool-wiki-gender-stats: Only keep detailled data of latest snapshot - https://phabricator.wikimedia.org/T433196#12196233 (10Danya) 05Stalled→03In progress
[18:02:47] <wikibugs>	 10Tool-wiki-gender-stats: Content-Security-Policy error when loading OSM map tiles - https://phabricator.wikimedia.org/T432531#12196236 (10Danya) p:05Low→03Lowest
[18:18:33] <wikibugs>	 (03update) 10andrew: Add v6 PTR for mail.wikimedia.az [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/343 (https://phabricator.wikimedia.org/T432761)
[18:18:38] <wikibugs>	 (03update) 10andrew: Add v6 PTR for mail.wikimedia.az [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/343 (https://phabricator.wikimedia.org/T432761)
[18:22:20] <wikibugs>	 10Tool-wiki-gender-stats: Non-Wikipedia wikis categorized as Wikipedia language codes - https://phabricator.wikimedia.org/T432474#12196269 (10Danya) 05In progress→03Resolved
[18:23:01] <wikibugs>	 10Tool-wiki-gender-stats: [TF] Only keep detailled data of latest snapshot - https://phabricator.wikimedia.org/T433196#12196271 (10Danya)
[18:26:38] <wikibugs>	 10Tool-wiki-gender-stats: [TF] Ingest data for August 2026 - https://phabricator.wikimedia.org/T434328 (10Danya) 03NEW
[18:26:40] <wikibugs>	 10Tool-wiki-gender-stats: [TF] Ingest data for August 2026 - https://phabricator.wikimedia.org/T434328#12196287 (10Danya) p:05Triage→03High
[18:26:57] <wikibugs>	 10Tool-wiki-gender-stats: [TF] Ingest data for August 2026 - https://phabricator.wikimedia.org/T434328#12196289 (10Danya)
[18:29:02] <wikibugs>	 (03open) 10danyya: Draft: Resolve T433196 "[TF] Only keep detailled data of latest snapshot" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/98
[19:12:38] <wikibugs>	 06tools-infrastructure-team, 10Continuous-Integration-Infrastructure, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, and 2 others: cloudcephosd1043 drives are very very busy - https://phabricator.wikimedia.org/T434334 (10Andrew) 03NEW
[19:15:34] <wikibugs>	 06tools-infrastructure-team, 10Continuous-Integration-Infrastructure, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, and 2 others: cloudcephosd1043 drives are very very busy - https://phabricator.wikimedia.org/T434334#12196432 (10Andrew) p:05Triage→03High a:03Andrew This started happening no...
[19:17:07] <wikibugs>	 10Cloud-VPS, 06tools-infrastructure-team: cloudcephosd1043 drives are very very busy - https://phabricator.wikimedia.org/T434334#12196437 (10JJMC89)
[19:18:12] <wikibugs>	 (03update) 10danyya: Draft: Resolve T433196 "[TF] Only keep detailled data of latest snapshot" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/98
[19:20:16] <wikibugs>	 (03update) 10danyya: Resolve T433196 "[TF] Only keep detailled data of latest snapshot" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/98
[19:20:21] <wikibugs>	 (03merge) 10danyya: Resolve T433196 "[TF] Only keep detailled data of latest snapshot" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/98
[19:33:24] <wikibugs>	 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387#12196457 (10Andrew) >>! In T429387#12154524, @Andrew wrote: > I am setting >  >  > ` > root@cloudcephmon1005:~# ceph confi...
[19:41:44] <jinxer-wm>	 FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors
[19:46:53] <wikibugs>	 10Striker, 06tools-platform-team, 10Diffusion, 10Phabricator, and 2 others: Striker: Always set all IO to none and hidden for internal URIs when creating Diffusion repositories - https://phabricator.wikimedia.org/T409341#12196487 (10Aklapper)
[19:47:16] <wikibugs>	 (03CR) 10Aklapper: "This should be ready for another review." [labs/striker] - 10https://gerrit.wikimedia.org/r/1315945 (https://phabricator.wikimedia.org/T409341) (owner: 10Aklapper)
[20:05:04] <wikibugs>	 10Tools: QuickStatements anti-abuse measure (rate limit?) - Cannot automatically assign ID - https://phabricator.wikimedia.org/T350262#12196566 (10Aklapper) 05Open→03Invalid Closing in favor of https://github.com/magnusmanske/quickstatements/issues/51 as that's where the tool's issue tracker is located (...
[20:12:02] <jinxer-wm>	 FIRING: [4x] PuppetZeroResources: Puppet has failed generate resources on cloudcephosd1023:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[20:13:57] <jinxer-wm>	 FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudcumin1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[20:17:15] <jinxer-wm>	 FIRING: [32x] PuppetZeroResources: Puppet has failed generate resources on cloudcephmon1006:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[20:22:34] <jinxer-wm>	 FIRING: [52x] PuppetZeroResources: Puppet has failed generate resources on cloudcephmon1006:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[20:27:16] <jinxer-wm>	 FIRING: [73x] PuppetZeroResources: Puppet has failed generate resources on cloudbackup2003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[20:28:29] <jinxer-wm>	 FIRING: [2x] PuppetZeroResources: Puppet has failed generate resources on cloudcumin1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[20:28:35] <wikibugs>	 06tools-infrastructure-team, 06SRE: openjdk upgrade breaks puppetservers - https://phabricator.wikimedia.org/T434339 (10Andrew) 03NEW
[20:28:54] <wikibugs>	 06tools-infrastructure-team, 06SRE: openjdk upgrade breaks puppetservers - https://phabricator.wikimedia.org/T434339#12196607 (10Andrew) p:05Triage→03Unbreak!
[20:31:44] <jinxer-wm>	 RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors
[20:32:25] <jinxer-wm>	 FIRING: [97x] PuppetZeroResources: Puppet has failed generate resources on cloudbackup2003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[20:37:43] <jinxer-wm>	 FIRING: [119x] PuppetZeroResources: Puppet has failed generate resources on cloudbackup1001-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[20:40:25] <wikibugs>	 06tools-infrastructure-team, 06SRE: openjdk upgrade breaks puppetservers - https://phabricator.wikimedia.org/T434339#12196637 (10Andrew) 05Open→03Resolved a:03Andrew @taavi observed that my dependency-chasing was in vain and that we just needed to restart the puppetservers. @jhathaway upgraded the jd...
[20:43:15] <jinxer-wm>	 FIRING: [134x] PuppetZeroResources: Puppet has failed generate resources on cloudbackup1001-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[20:47:38] <jinxer-wm>	 FIRING: [134x] PuppetZeroResources: Puppet has failed generate resources on cloudbackup1001-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[20:53:10] <jinxer-wm>	 RESOLVED: [59x] PuppetZeroResources: Puppet has failed generate resources on cloudbackup1002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[21:15:18] <jinxer-wm>	 RESOLVED: PuppetZeroResources: Puppet has failed generate resources on cloudcumin2001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[22:32:06] <wikibugs>	 10Tool-wiki-gender-stats: [TF] Compute aggregated stats for each wiki project version - https://phabricator.wikimedia.org/T433199#12196839 (10Danya) 05In progress→03Resolved
[22:51:19] <wikibugs>	 10Striker, 06tools-platform-team, 07Documentation: Expose default documentation space in Wikitech for Toolforge tools in ToolsAdmin and/or Toolhub - https://phabricator.wikimedia.org/T433325#12196861 (10bd808) {T123429} has been there for 10 years. https://wikitech.wikimedia.org/wiki/Category:Toolforge_tools...