[00:35:40] <icinga-wm>	 PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static
[00:40:38] <icinga-wm>	 RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 30033 bytes in 8.111 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static
[00:43:47] <wikibugs>	 (03update) 10system625: feat: Add SPARQL query link to author's works list page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/81
[00:46:40] <icinga-wm>	 PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static
[00:53:38] <icinga-wm>	 RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 30031 bytes in 9.369 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static
[00:56:40] <icinga-wm>	 PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static
[01:00:30] <icinga-wm>	 RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 30030 bytes in 0.358 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static
[01:52:40] <icinga-wm>	 PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static
[01:54:38] <icinga-wm>	 RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 30031 bytes in 8.939 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static
[01:57:40] <icinga-wm>	 PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static
[02:01:38] <icinga-wm>	 RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 30033 bytes in 8.687 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static
[02:15:39] <wikibugs>	 10Tool-wsindex, 10Developer-Outreach, 10Wikisource Reader App, 10Outreachy (Round 31): Outreachy 31: Improve the Wikisource Reader App - https://phabricator.wikimedia.org/T405593#11310455 (10Bodhisattwa)
[03:57:26] <wikibugs>	 10Tool-paulina, 10Outreachy (Round 31): Outreachy 31: Features to edit author and work data on Wikidata directly from Paulina - https://phabricator.wikimedia.org/T392429#11310478 (10Nat_WDU) Hello @Kimbrene-kakande I'm proritizing the review for PRs related to the listed microtasks. Thanks for understanding,
[08:21:40] <icinga-wm>	 PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static
[08:22:34] <icinga-wm>	 RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 30031 bytes in 5.598 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static
[09:22:35] <wikibugs>	 (03update) 10miiswom: Proof of concept for adding new statements [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/153
[09:23:32] <wikibugs>	 06cloud-services-team (FY2025/26-Q1), 10Cloud-VPS, 06DC-Ops, 06SRE: Experiment with cloudcephosd1050 and cloudcephosd1051 in single-nic configuration - https://phabricator.wikimedia.org/T405478#11310822 (10fgiunchedi) 05Open→03Resolved This is complete, both hosts are in service with full weight an...
[09:24:19] <wikibugs>	 06cloud-services-team (FY2025/26-Q1), 10Cloud-VPS, 06DC-Ops, 06SRE: cloudcephosd10[48-52] service implementation - https://phabricator.wikimedia.org/T395910#11310826 (10fgiunchedi)
[09:24:25] <wikibugs>	 06cloud-services-team (FY2025/26-Q1), 10Cloud-VPS, 06DC-Ops, 06SRE: cloudcephosd10[48-52] service implementation - https://phabricator.wikimedia.org/T395910#11310828 (10fgiunchedi) 05In progress→03Resolved Completed
[09:30:20] <wikibugs>	 (03approved) 10fnegri: builds-builder: add script to upgrade taskruns [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1010 (https://phabricator.wikimedia.org/T408127) (owner: 10dcaro)
[09:31:17] <wikibugs>	 (03update) 10dcaro: builds-builder: add script to upgrade taskruns [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1010 (https://phabricator.wikimedia.org/T408127)
[09:40:56] <wikibugs>	 06cloud-services-team, 10Toolforge, 07Documentation: [docs] update READMEs - https://phabricator.wikimedia.org/T362390#11310873 (10dcaro) →14Duplicate dup:03T407477
[09:40:59] <wikibugs>	 10Toolforge (Toolforge iteration 24): [docs] update all readmes with the same deployment docs - https://phabricator.wikimedia.org/T407477#11310876 (10dcaro)
[09:41:29] <wikibugs>	 06cloud-services-team, 10Toolforge: Policy for (external) contributions / code style - https://phabricator.wikimedia.org/T408331#11310880 (10dcaro) Related tasks {T392524}, {T394595}, {T407477}
[09:42:33] <wikibugs>	 (03update) 10dcaro: builds-builder: add script to upgrade taskruns [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1010 (https://phabricator.wikimedia.org/T408127)
[09:45:21] <wikibugs>	 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [builds-builder] Host all the build-time dependencies on gitlab - https://phabricator.wikimedia.org/T408140#11310930 (10dcaro) p:05Triage→03Medium
[09:46:08] <wikibugs>	 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [builds-builder] Host all the build-time dependencies on gitlab - https://phabricator.wikimedia.org/T408140#11310932 (10dcaro) For context, last friday github rate-limited us and I was failing to run the tests in prod/toolsbeta/local due to this.
[09:46:36] <wikibugs>	 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [builds-builder] Host all the build-time dependencies on gitlab - https://phabricator.wikimedia.org/T408140#11310933 (10dcaro) a:03dcaro
[09:46:47] <wikibugs>	 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [builds-api, maintain-harbor] fix build/image cleanup - https://phabricator.wikimedia.org/T404157#11310936 (10dcaro) p:05Triage→03Medium
[09:46:52] <wikibugs>	 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [builds-api, maintain-harbor] fix build/image cleanup - https://phabricator.wikimedia.org/T404157#11310937 (10dcaro) p:05Medium→03Triage
[09:47:08] <wikibugs>	 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [builds-api, maintain-harbor] fix build/image cleanup - https://phabricator.wikimedia.org/T404157#11310938 (10dcaro) p:05Triage→03Medium
[09:47:42] <wikibugs>	 10Toolforge (Toolforge iteration 24): [builds-api] listing deployments gets very slow when querying taskruns - https://phabricator.wikimedia.org/T408125#11310954 (10dcaro) 05Open→03In progress a:03dcaro
[09:47:53] <wikibugs>	 10Toolforge (Toolforge iteration 24): [docs] update all readmes with the same deployment docs - https://phabricator.wikimedia.org/T407477#11310957 (10dcaro) a:03dcaro
[09:49:12] <wikibugs>	 06cloud-services-team, 10Toolforge: Allow toolforge jobs to be scheduled based on local time in specific time zones. - https://phabricator.wikimedia.org/T408371 (10Tkarcher) 03NEW
[09:53:00] <wikibugs>	 06cloud-services-team, 10Toolforge: [jobs-api] Allow toolforge jobs to be scheduled based on local time in specific time zones. - https://phabricator.wikimedia.org/T408371#11311007 (10dcaro)
[09:56:42] <wikibugs>	 (03update) 10dcaro: builds-builder: add script to upgrade taskruns [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1010 (https://phabricator.wikimedia.org/T408127)
[09:57:43] <wikibugs>	 06cloud-services-team: DiskSpace Disk space cloudbackup1002-dev:9100:/srv/cinder-backups 0% free - https://phabricator.wikimedia.org/T407420#11311051 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi
[09:57:53] <wikibugs>	 06cloud-services-team: DiskSpace Disk space cloudbackup1002-dev:9100:/srv/cinder-backups 0% free - https://phabricator.wikimedia.org/T407420#11311054 (10fgiunchedi) FS space is back
[09:58:06] <wikibugs>	 06cloud-services-team: PuppetFailure Puppet has failed on cloudbackup1002-dev:9100 - https://phabricator.wikimedia.org/T407429#11311059 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi Space is back
[09:59:13] <wikibugs>	 06cloud-services-team: SystemdUnitDown The systemd unit backup_cinder_volumes.service on node cloudbackup1002-dev has been failing for more than two hours. - https://phabricator.wikimedia.org/T407437#11311069 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi fs space is back
[09:59:21] <wikibugs>	 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T407438#11311073 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi fs space is back
[09:59:30] <wikibugs>	 06cloud-services-team: NodeDown Node cloudbackup1002-dev has been down for long. - https://phabricator.wikimedia.org/T407456#11311086 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi fs space is back
[09:59:40] <wikibugs>	 06cloud-services-team: SystemdUnitDown The systemd unit postgresql@15-main.service on node cloudbackup1002-dev has been failing for more than two hours. - https://phabricator.wikimedia.org/T407524#11311090 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi fs space is back
[10:01:19] <wikibugs>	 06cloud-services-team: JobUnavailable Reduced availability for job openstack in cloud@codfw - https://phabricator.wikimedia.org/T407470#11311099 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi Alert is gone, optimistically resolving
[10:01:25] <wikibugs>	 06cloud-services-team: NovafullstackSustainedFailures Novafullstack tests have been failing for more than 5hours in eqiad - https://phabricator.wikimedia.org/T407683#11311103 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi
[10:01:34] <wikibugs>	 06cloud-services-team: JobUnavailable Reduced availability for job openstack in cloud@eqiad - https://phabricator.wikimedia.org/T407688#11311105 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi
[10:01:53] <wikibugs>	 06cloud-services-team: JobUnavailable Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://phabricator.wikimedia.org/T407868#11311109 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi Alert is gone, optimistically resolving
[10:03:17] <wikibugs>	 06cloud-services-team, 06SRE: latest Trixie image (as of 2025-10-16) grub failure on R450 hardware - https://phabricator.wikimedia.org/T407586#11311121 (10fgiunchedi) p:05Triage→03High
[10:03:47] <wikibugs>	 06cloud-services-team, 10Toolforge: [components-api] tries to decode server errors as json - https://phabricator.wikimedia.org/T408028#11311122 (10fgiunchedi) p:05Triage→03Medium
[10:03:56] <wikibugs>	 06cloud-services-team, 10Toolforge: toolforge jobs dump includes booleans as strings - https://phabricator.wikimedia.org/T408034#11311123 (10fgiunchedi) p:05Triage→03Medium
[10:04:07] <wikibugs>	 06cloud-services-team, 10Toolforge: [functional tests,toolforge-deploy] functional tests are optimistic about retries/timeouts - https://phabricator.wikimedia.org/T408002#11311124 (10fgiunchedi) p:05Triage→03Medium
[10:04:23] <wikibugs>	 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Persistence, and 3 others: Set up x1 replication to an-redacteddb1001 - https://phabricator.wikimedia.org/T407485#11311125 (10fgiunchedi) p:05Triage→03Medium
[10:04:27] <wikibugs>	 06cloud-services-team, 10Toolforge: [build service] Python pack is outdated, does not support stable release - https://phabricator.wikimedia.org/T408108#11311126 (10fgiunchedi) p:05Triage→03Medium
[10:04:40] <wikibugs>	 06cloud-services-team, 10Toolforge: [builds] What are the current best practices for CI? - https://phabricator.wikimedia.org/T408321#11311127 (10fgiunchedi) p:05Triage→03Medium
[10:04:53] <wikibugs>	 06cloud-services-team, 10Quarry, 06RoadToWiki, 13Patch-For-Review: quarry: Use a proper Python package manager - https://phabricator.wikimedia.org/T397332#11311128 (10fgiunchedi) p:05Triage→03Medium
[10:04:57] <wikibugs>	 06cloud-services-team, 10Data-Services: [wikireplicas] Create views for new wiki minwikisource - https://phabricator.wikimedia.org/T408346#11311129 (10fgiunchedi) p:05Triage→03Medium
[10:05:06] <wikibugs>	 06cloud-services-team, 10Data-Services: [wikireplicas] Create views for new wiki pcmwikiquote - https://phabricator.wikimedia.org/T408354#11311130 (10fgiunchedi) p:05Triage→03Medium
[10:05:17] <wikibugs>	 06cloud-services-team, 10Toolforge: Policy for (external) contributions / code style - https://phabricator.wikimedia.org/T408331#11311131 (10fgiunchedi) p:05Triage→03Medium
[10:05:26] <wikibugs>	 06cloud-services-team, 10Toolforge: [jobs-api] Allow toolforge jobs to be scheduled based on local time in specific time zones. - https://phabricator.wikimedia.org/T408371#11311132 (10fgiunchedi) p:05Triage→03Medium
[10:05:51] <wikibugs>	 (03update) 10miiswom: Proof of concept for adding new statements [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/153
[10:21:46] <wikibugs>	 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [bulids-builder,tekton] upgrade taskuns to v1 from v1beta1 in storage and delete v1beta1 from stored versions - https://phabricator.wikimedia.org/T408127#11311197 (10dcaro) 05Open→03In progress
[10:21:53] <wikibugs>	 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [bulids-builder,tekton] upgrade taskuns to v1 from v1beta1 in storage and delete v1beta1 from stored versions - https://phabricator.wikimedia.org/T408127#11311199 (10dcaro) a:03dcaro
[10:53:56] <wikibugs>	 06cloud-services-team, 10Toolforge: [builds] What are the current best practices for CI? - https://phabricator.wikimedia.org/T408321#11311303 (10dcaro) > Which leads to the question of what is the current best practice for testing build pack based images?  Short answer, there's no current best practice, that i...
[10:59:08] <wikibugs>	 10Tool-global-search: Even if I change the language it doesn't reflect - https://phabricator.wikimedia.org/T363335#11311318 (10jhsoby) →14Duplicate dup:03T366410
[10:59:16] <wikibugs>	 10Tool-global-search: Global Search: Language selector doesn't work and 7 languages ​​have no labels - https://phabricator.wikimedia.org/T366410#11311320 (10jhsoby)
[10:59:26] <wikibugs>	 (03approved) 10dcaro: builds-builder: add script to upgrade taskruns [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1010 (https://phabricator.wikimedia.org/T408127)
[10:59:37] <wikibugs>	 (03update) 10dcaro: builds-builder: add script to upgrade taskruns [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1010 (https://phabricator.wikimedia.org/T408127)
[11:00:26] <wikibugs>	 (03merge) 10dcaro: builds-builder: add script to upgrade taskruns [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1010 (https://phabricator.wikimedia.org/T408127)
[11:00:56] <wikibugs>	 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [bulids-builder,tekton] upgrade taskuns to v1 from v1beta1 in storage and delete v1beta1 from stored versions - https://phabricator.wikimedia.org/T408127#11311332 (10dcaro) Updating the stored version improved considerably the time it takes to get a tas...
[11:13:10] <wikibugs>	 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [bulids-builder,tekton] upgrade taskuns to v1 from v1beta1 in storage and delete v1beta1 from stored versions - https://phabricator.wikimedia.org/T408127#11311360 (10dcaro) Patched toolsbeta, no clear difference, running tests now: ` root@toolsbeta-test...
[11:14:21] <wikibugs>	 (03update) 10miiswom: Proof of concept for adding new statements [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/153
[11:19:13] <wikibugs>	 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [bulids-builder,tekton] upgrade taskuns to v1 from v1beta1 in storage and delete v1beta1 from stored versions - https://phabricator.wikimedia.org/T408127#11311371 (10dcaro) Deployed in tools, no big difference either :/, running tests: ` root@tools-k8s-...
[11:24:14] <wikibugs>	 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [bulids-builder,tekton] upgrade taskuns to v1 from v1beta1 in storage and delete v1beta1 from stored versions - https://phabricator.wikimedia.org/T408127#11311373 (10dcaro) Completely disabling the webhook in the crd did not change anything in toolsbeta...
[11:27:42] <wikibugs>	 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [bulids-builder,tekton] upgrade taskuns to v1 from v1beta1 in storage and delete v1beta1 from stored versions - https://phabricator.wikimedia.org/T408127#11311380 (10dcaro) Getting a single taskrun by name is pretty fast though (<0.2 seconds in prod, vs...
[11:29:56] <wikibugs>	 10Toolforge (Toolforge iteration 24): [builds-api] Fetch taskruns from the pipelinerun object `status.childReferences` field - https://phabricator.wikimedia.org/T408384 (10dcaro) 03NEW
[11:43:15] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 10Toolforge: CluodVPS instance for ProVe - https://phabricator.wikimedia.org/T408387 (10Albert.meronyo) 03NEW
[11:44:02] <wikibugs>	 10superset.wmcloud.org, 10PendingChangesBot, 10Pywikibot, 10superset.wikimedia.org: Pywikibot OAuth/BotPassword authentication fails when login to third-party Wikimedia sites (Superset, Commons Query) - https://phabricator.wikimedia.org/T408286#11311451 (10Xqt)
[11:45:47] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 10Toolforge: CloudVPS instance for ProVe - https://phabricator.wikimedia.org/T408387#11311456 (10LucasWerkmeister)
[11:46:50] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 10Toolforge: CloudVPS instance for ProVe - https://phabricator.wikimedia.org/T408387#11311457 (10LucasWerkmeister)
[11:47:45] <wikibugs>	 06cloud-services-team, 10superset.wmcloud.org, 10PendingChangesBot, 10Pywikibot: Pywikibot OAuth/BotPassword authentication fails when login to third-party Wikimedia sites (Superset, Commons Query) - https://phabricator.wikimedia.org/T408286#11311459 (10Xqt)
[12:20:38] <wikibugs>	 10Tool-paulina: Proof of concept: add work form, with autocomplete - https://phabricator.wikimedia.org/T407558#11311561 (10Adwivedii) Hello @Pepe_piton @Nat_WDU,  Kindly review my MR for this task: [[ https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/155 ]]   Thank you!!
[12:28:38] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Project-requests): CloudVPS instance for ProVe - https://phabricator.wikimedia.org/T408387#11311574 (10JJMC89)
[12:41:30] <wikibugs>	 (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/49
[12:41:30] <wikibugs>	 (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/19
[12:43:55] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/1198967 (owner: 10L10n-bot)
[12:44:02] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/1198971 (owner: 10L10n-bot)
[12:58:57] <wikibugs>	 (03update) 10raymond-ndibe: [jobs.py] breakup get_tags_in_use_by_toolforge_deploy gitlab querying into chunks [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/54 (https://phabricator.wikimedia.org/T407496)
[12:59:01] <wikibugs>	 (03update) 10raymond-ndibe: [jobs.py] breakup get_tags_in_use_by_toolforge_deploy gitlab querying into chunks [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/54 (https://phabricator.wikimedia.org/T407496)
[13:02:39] <wikibugs>	 (03update) 10raymond-ndibe: [jobs.py] breakup get_tags_in_use_by_toolforge_deploy gitlab querying into chunks [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/54 (https://phabricator.wikimedia.org/T407496)
[13:15:51] <wikibugs>	 (03update) 10raymond-ndibe: [jobs.py] breakup get_tags_in_use_by_toolforge_deploy gitlab querying into chunks [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/54 (https://phabricator.wikimedia.org/T407496)
[13:18:08] <wikibugs>	 (03open) 10devjadiya: Initial commit - Django project setup [toolforge-repos/script-publisher] - 10https://gitlab.wikimedia.org/toolforge-repos/script-publisher/-/merge_requests/1
[13:18:15] <wikibugs>	 (03merge) 10devjadiya: Initial commit - Django project setup [toolforge-repos/script-publisher] - 10https://gitlab.wikimedia.org/toolforge-repos/script-publisher/-/merge_requests/1
[13:21:44] <wikibugs>	 (03update) 10raymond-ndibe: [jobs.py] breakup get_tags_in_use_by_toolforge_deploy gitlab querying into chunks [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/54 (https://phabricator.wikimedia.org/T407496)
[13:31:34] <wikibugs>	 (03approved) 10raymond-ndibe: [jobs.py] breakup get_tags_in_use_by_toolforge_deploy gitlab querying into chunks [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/54 (https://phabricator.wikimedia.org/T407496)
[13:31:36] <wikibugs>	 (03update) 10raymond-ndibe: [jobs.py] breakup get_tags_in_use_by_toolforge_deploy gitlab querying into chunks [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/54 (https://phabricator.wikimedia.org/T407496)
[13:31:43] <wikibugs>	 (03merge) 10raymond-ndibe: [jobs.py] breakup get_tags_in_use_by_toolforge_deploy gitlab querying into chunks [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/54 (https://phabricator.wikimedia.org/T407496)
[13:34:32] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: maintain-harbor: bump to 0.0.61-20251027133151-77adbf83 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1012 (https://phabricator.wikimedia.org/T407496)
[13:49:15] <wikibugs>	 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Make sure the Wikimedia REST API OpenAPI spec passes the base OpenAPI linting ruleset - https://phabricator.wikimedia.org/T406914#11311837 (10BPirkle) Took a quick look at this, and my first impression is that it will be a nontrivial amount of work...
[13:55:24] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
[13:57:50] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
[13:58:24] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
[14:00:13] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
[14:08:12] <wikibugs>	 (03open) 10raymond-ndibe: [jobs.py] reduce chunk size to 2 in get_tag_in_use_by_toolforge_deploy [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/55 (https://phabricator.wikimedia.org/T407496)
[14:08:36] <wikibugs>	 (03approved) 10raymond-ndibe: [jobs.py] reduce chunk size to 2 in get_tag_in_use_by_toolforge_deploy [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/55 (https://phabricator.wikimedia.org/T407496)
[14:08:42] <wikibugs>	 (03update) 10raymond-ndibe: [jobs.py] reduce chunk size to 2 in get_tag_in_use_by_toolforge_deploy [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/55 (https://phabricator.wikimedia.org/T407496)
[14:10:11] <wikibugs>	 (03merge) 10raymond-ndibe: [jobs.py] reduce chunk size to 2 in get_tag_in_use_by_toolforge_deploy [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/55 (https://phabricator.wikimedia.org/T407496)
[14:10:53] <wikibugs>	 (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/49 (owner: 10l10n-bot)
[14:10:56] <wikibugs>	 (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/49 (owner: 10l10n-bot)
[14:12:48] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: maintain-harbor: bump to 0.0.62-20251027141017-cff43eaa [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1012 (https://phabricator.wikimedia.org/T407496)
[14:16:07] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor
[14:18:29] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor
[14:23:06] <wikibugs>	 (03open) 10dcaro: build.getTaskStatusRunForPipeline: get taskrun name from pipeline data [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/147 (https://phabricator.wikimedia.org/T408384)
[14:49:57] <wikibugs>	 (03update) 10dcaro: build.getTaskStatusRunForPipeline: get taskrun name from pipeline data [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/147 (https://phabricator.wikimedia.org/T408384)
[15:04:27] <wikibugs>	 (03approved) 10dcaro: build.getTaskStatusRunForPipeline: get taskrun name from pipeline data [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/147 (https://phabricator.wikimedia.org/T408384)
[15:04:41] <wikibugs>	 (03merge) 10dcaro: build.getTaskStatusRunForPipeline: get taskrun name from pipeline data [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/147 (https://phabricator.wikimedia.org/T408384)
[15:14:03] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-api: bump to 0.0.203-20251027150453-91f95b0f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1009 (https://phabricator.wikimedia.org/T403322 https://phabricator.wikimedia.org/T408384)
[15:14:08] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-api: bump to 0.0.203-20251027150453-91f95b0f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1009 (https://phabricator.wikimedia.org/T403322 https://phabricator.wikimedia.org/T408384)
[15:25:55] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api
[15:31:23] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
[15:33:54] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api
[15:35:37] <wikibugs>	 10VPS-project-Codesearch, 06collaboration-services: Codesearch: Ensure logrotate for /var/log/account/pacct - https://phabricator.wikimedia.org/T408234#11312485 (10LSobanski) p:05Triage→03Low
[15:38:47] <wikibugs>	 06cloud-services-team, 10Toolforge: [jobs-api] Allow toolforge jobs to be scheduled based on local time in specific time zones. - https://phabricator.wikimedia.org/T408371#11312495 (10bd808) Support for specifying a time zone for a CronJob was added upstream in Kubernetes v1.27 as `.spec.timeZone` [[https://ku...
[15:39:34] <wikibugs>	 06cloud-services-team, 10Toolforge: [jobs-api] Allow toolforge job schedule to specify a time zone - https://phabricator.wikimedia.org/T408371#11312506 (10bd808)
[15:39:44] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
[15:44:50] <wikibugs>	 10VPS-project-Codesearch, 06collaboration-services, 07Sustainability (Incident Followup): Codesearch: Ensure logrotate for /var/log/account/pacct - https://phabricator.wikimedia.org/T408234#11312546 (10Krinkle)
[16:02:51] <wikibugs>	 10VPS-project-Wikistats: Add pcmwikiquote to wikistats - https://phabricator.wikimedia.org/T408356#11312611 (10Dzahn) 05Open→03Stalled until T408318
[16:03:46] <wikibugs>	 10VPS-project-Wikistats: Add minwikisource to wikistats - https://phabricator.wikimedia.org/T408348#11312622 (10Dzahn) 05Open→03Stalled until T408317
[16:04:30] <wikibugs>	 (03approved) 10dcaro: builds-api: bump to 0.0.203-20251027150453-91f95b0f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1009 (https://phabricator.wikimedia.org/T403322 https://phabricator.wikimedia.org/T408384) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[16:05:49] <wikibugs>	 (03merge) 10dcaro: builds-api: bump to 0.0.203-20251027150453-91f95b0f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1009 (https://phabricator.wikimedia.org/T403322 https://phabricator.wikimedia.org/T408384) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[16:06:24] <wikibugs>	 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [builds-api] Fetch taskruns from the pipelinerun object `status.childReferences` field - https://phabricator.wikimedia.org/T408384#11312637 (10dcaro) 05Open→03Resolved p:05Triage→03High
[16:06:37] <wikibugs>	 10Toolforge (Toolforge iteration 24): [bulids-builder,tekton] upgrade taskuns to v1 from v1beta1 in storage and delete v1beta1 from stored versions - https://phabricator.wikimedia.org/T408127#11312641 (10dcaro) 05In progress→03Resolved
[16:07:32] <wikibugs>	 10Toolforge (Toolforge iteration 24): [builds-api] listing deployments gets very slow when querying taskruns - https://phabricator.wikimedia.org/T408125#11312645 (10dcaro) 05In progress→03Resolved This is fixed now, both: * upgrading the crds to the latest version (so it does not go through the conversio...
[16:11:50] <icinga-wm>	 PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static
[16:12:40] <icinga-wm>	 RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 30030 bytes in 0.252 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static
[16:19:18] <wikibugs>	 (03open) 10fnegri: Simplify argparse testing [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/8
[16:19:18] <wikibugs>	 (03update) 10fnegri: Simplify argparse testing [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/8
[16:19:19] <wikibugs>	 (03update) 10fnegri: Replace only views that need updating [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/9 (https://phabricator.wikimedia.org/T351637)
[16:19:20] <wikibugs>	 (03open) 10fnegri: Replace only views that need updating [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/9 (https://phabricator.wikimedia.org/T351637)
[16:19:23] <wikibugs>	 (03update) 10fnegri: Simplify argparse testing [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/8
[16:19:26] <wikibugs>	 (03update) 10fnegri: Replace only views that need updating [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/9 (https://phabricator.wikimedia.org/T351637)
[16:21:20] <wikibugs>	 (03update) 10fnegri: Simplify argparse testing [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/8
[16:21:20] <wikibugs>	 (03update) 10fnegri: Replace only views that need updating [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/9 (https://phabricator.wikimedia.org/T351637)
[16:21:52] <wikibugs>	 (03update) 10fnegri: Replace only views that need updating [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/9 (https://phabricator.wikimedia.org/T351637)
[16:21:56] <wikibugs>	 (03update) 10fnegri: Simplify argparse testing [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/8
[17:03:09] <wikibugs>	 (03approved) 10dcaro: Simplify argparse testing [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/8 (owner: 10fnegri)
[17:08:52] <wikibugs>	 (03merge) 10dcaro: restart deployment via template hash change [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/218 (owner: 10damian)
[17:11:44] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.426-20251027170901-9f4619b4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1013
[17:11:48] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.426-20251027170901-9f4619b4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1013
[17:11:57] <wikibugs>	 (03update) 10dcaro: offload job restart logic to jobs-api [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/133 (https://phabricator.wikimedia.org/T403321) (owner: 10damian)
[17:18:43] <wikibugs>	 (03update) 10dcaro: offload job restart logic to jobs-api [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/133 (https://phabricator.wikimedia.org/T403321) (owner: 10damian)
[17:19:36] <wikibugs>	 (03update) 10dcaro: api - add job_changed to update response [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/228 (https://phabricator.wikimedia.org/T403321) (owner: 10damian)
[17:19:51] <wikibugs>	 (03update) 10dcaro: api - add job_changed to update response [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/228 (https://phabricator.wikimedia.org/T403321) (owner: 10damian)
[17:41:04] <wikibugs>	 (03open) 10dcaro: toolforge_deploy_mr.py: show the commands being run [repos/cloud/toolforge/lima-kilo] (move_local_deploy_to_deploy_mr) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/291
[17:42:23] <wikibugs>	 (03update) 10dcaro: toolforge_deploy_mr.py: show the commands being run [repos/cloud/toolforge/lima-kilo] (move_local_deploy_to_deploy_mr) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/291
[17:51:47] <wikibugs>	 (03update) 10dcaro: toolforge_deploy_mr: support deploying local code [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/290
[17:52:54] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
[17:53:35] <wikibugs>	 (03open) 10dcaro: readme: standardize deployment docs [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/29
[17:57:44] <wikibugs>	 (03update) 10dcaro: readme: standardize deployment docs [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/29
[17:58:01] <wikibugs>	 (03update) 10dcaro: readme: standardize deployment docs [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/29
[18:01:00] <wikibugs>	 (03update) 10dcaro: README: update deployment instructions [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/230
[18:01:25] <wikibugs>	 (03update) 10dcaro: README: update deployment instructions [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/230
[18:01:42] <wikibugs>	 (03update) 10dcaro: toolforge_deploy: support deploying local code [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/290
[18:02:04] <wikibugs>	 (03update) 10dcaro: toolforge_deploy: support deploying local code [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/290
[18:05:10] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
[18:05:37] <wikibugs>	 (03update) 10dcaro: README: update deployment instructions [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/230
[18:05:45] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
[18:09:03] <wikibugs>	 (03update) 10dcaro: toolforge_deploy_mr.py: show the commands being run [repos/cloud/toolforge/lima-kilo] (move_local_deploy_to_deploy_mr) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/291
[18:18:17] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
[18:22:45] <wikibugs>	 (03approved) 10dcaro: jobs-api: bump to 0.0.426-20251027170901-9f4619b4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1013 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[18:22:48] <wikibugs>	 (03merge) 10dcaro: jobs-api: bump to 0.0.426-20251027170901-9f4619b4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1013 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[18:27:49] <wikibugs>	 (03open) 10damian: Remove `job` from `JobsJobResponse` [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/146
[18:32:50] <wikibugs>	 (03update) 10damian: Remove `job` from `JobsJobResponse` [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/146
[18:34:28] <wikibugs>	 (03update) 10damian: offload job restart logic to jobs-api [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/133 (https://phabricator.wikimedia.org/T403321)
[18:35:06] <wikibugs>	 (03update) 10damian: offload job restart logic to jobs-api [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/133 (https://phabricator.wikimedia.org/T403321)
[18:35:21] <wikibugs>	 (03update) 10damian: offload job restart logic to jobs-api [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/133 (https://phabricator.wikimedia.org/T403321)
[18:35:29] <wikibugs>	 (03update) 10damian: Remove `job` from `JobsJobResponse` [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/146 (https://phabricator.wikimedia.org/T403321)
[18:42:43] <wikibugs>	 (03update) 10damian: api - add job_changed to update response [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/228 (https://phabricator.wikimedia.org/T403321)
[18:45:03] <wikibugs>	 (03update) 10damian: api - add job_changed to update response [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/228 (https://phabricator.wikimedia.org/T403321)
[18:51:06] <wikibugs>	 (03update) 10damian: offload job restart logic to jobs-api [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/133 (https://phabricator.wikimedia.org/T403321)
[18:54:26] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
[18:55:44] <wikibugs>	 (03update) 10damian: offload job restart logic to jobs-api [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/133 (https://phabricator.wikimedia.org/T403321)
[18:56:34] <wikibugs>	 (03update) 10damian: offload job restart logic to jobs-api [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/133 (https://phabricator.wikimedia.org/T403321)
[18:58:08] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
[19:00:08] <wikibugs>	 (03update) 10damian: offload job restart logic to jobs-api [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/133 (https://phabricator.wikimedia.org/T403321)
[19:08:17] <jinxer-wm>	 FIRING: JobUnavailable: Reduced availability for job rabbitmq in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[19:13:17] <jinxer-wm>	 FIRING: [2x] JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[19:13:39] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
[19:15:49] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for all services
[19:16:05] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
[19:16:18] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for all services
[19:16:50] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
[19:17:59] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for all services
[19:48:17] <jinxer-wm>	 RESOLVED: JobUnavailable: Reduced availability for job rabbitmq in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[20:04:17] <jinxer-wm>	 FIRING: JobUnavailable: Reduced availability for job rabbitmq in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[20:09:17] <jinxer-wm>	 RESOLVED: JobUnavailable: Reduced availability for job rabbitmq in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[20:10:48] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
[20:14:58] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
[20:16:38] <wikibugs>	 10Tool-gitlab-bot, 10Wikibugs, 10GitLab (Integrations): Automate setup of comment, pipeline, and job webhooks for all GitLab projects - https://phabricator.wikimedia.org/T362940#11313650 (10bd808)
[20:43:39] <wikibugs>	 10Tool-gitlab-bot: Create a skeltal GitLab bot that can respond to messages in merge request comments - https://phabricator.wikimedia.org/T408445 (10bd808) 03NEW
[21:01:57] <icinga-wm>	 PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static
[21:08:49] <icinga-wm>	 RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 30030 bytes in 0.262 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static
[21:30:47] <wikibugs>	 06cloud-services-team (FY2017-18), 06Infrastructure-Foundations, 06SRE, 07Puppet: Upgrade to puppet 4 (4.8 or newer) - https://phabricator.wikimedia.org/T177254#11313878 (10Krinkle)
[21:42:55] <wikibugs>	 06cloud-services-team (FY2017-18), 06Infrastructure-Foundations, 06SRE, 07Puppet: Upgrade to puppet 4 (4.8 or newer) - https://phabricator.wikimedia.org/T177254#11314006 (10Krinkle)
[21:58:51] <wikibugs>	 06cloud-services-team, 10Toolforge: s53734__toolviews_p not updated since 2025-10-22 - https://phabricator.wikimedia.org/T408454 (10bd808) 03NEW
[22:02:11] <wikibugs>	 06cloud-services-team, 10Toolforge: Toolviews table (s53734__toolviews_p) not updated since 2025-10-22 - https://phabricator.wikimedia.org/T408454#11314134 (10bd808)
[22:08:19] <wikibugs>	 06cloud-services-team, 10Toolforge: Toolviews table (s53734__toolviews_p) not updated since 2025-10-22 - https://phabricator.wikimedia.org/T408454#11314191 (10taavi) p:05Triage→03High a:03taavi Hmh, I should have checked this after making the change. Anyway, it seems like the issue is a Hiera issue somew...
[22:17:17] <jinxer-wm>	 FIRING: JobUnavailable: Reduced availability for job rabbitmq in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[22:18:00] <wikibugs>	 10Toolforge, 06tools-infrastructure-team: Add monitoring to check that toolviews process is successfully working - https://phabricator.wikimedia.org/T408457 (10taavi) 03NEW p:05Triage→03Medium
[22:18:11] <wikibugs>	 06cloud-services-team, 10Toolforge: Toolviews table (s53734__toolviews_p) not updated since 2025-10-22 - https://phabricator.wikimedia.org/T408454#11314286 (10taavi) 05Open→03Resolved Tracking further improvements in T408457.
[22:22:17] <jinxer-wm>	 FIRING: [2x] JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[23:06:08] <wikibugs>	 10Tool-paulina: Proof of concept: add work form, with autocomplete - https://phabricator.wikimedia.org/T407558#11314468 (10Pepe_piton) @Rolalove Good work! Autocomplete works very well.  I also liked the way you handled the styles of the autocomplete list! Just a couple of comments. Perhaps it would be better to...
[23:22:17] <jinxer-wm>	 FIRING: [2x] JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[23:24:48] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on cloudrabbit2003-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[23:27:17] <jinxer-wm>	 FIRING: [2x] JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[23:32:17] <jinxer-wm>	 FIRING: [2x] JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[23:37:17] <jinxer-wm>	 FIRING: [2x] JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[23:39:48] <jinxer-wm>	 FIRING: [2x] PuppetFailure: Puppet has failed on cloudrabbit2001-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[23:47:17] <jinxer-wm>	 FIRING: [2x] JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[23:52:17] <jinxer-wm>	 FIRING: [2x] JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[23:52:37] <wikibugs>	 10Tool-paulina: Proof of concept: add work form, with autocomplete - https://phabricator.wikimedia.org/T407558#11315435 (10Oluwatumininu.m)    > @Oluwatumininu.m Thanks, very good contribution! I really liked the work you did giving priority to the autocomplete suggestions that are actually types of works. >  Th...