[00:08:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[01:11:03] <jinxer-wm>	 FIRING: [3x] PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol2005-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange
[02:42:19] <jinxer-wm>	 FIRING: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling
[03:02:16] <wikibugs>	 (03update) 10raymond-ndibe: [tests] account for warning messages printed to stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/883 (https://phabricator.wikimedia.org/T400390)
[03:02:19] <jinxer-wm>	 RESOLVED: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling
[03:02:28] <wikibugs>	 (03update) 10raymond-ndibe: [tests] account for warning messages printed to stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/883 (https://phabricator.wikimedia.org/T400390)
[03:05:07] <wikibugs>	 (03update) 10raymond-ndibe: [tests] account for warning messages printed to stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/883 (https://phabricator.wikimedia.org/T400390)
[03:12:19] <jinxer-wm>	 FIRING: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling
[03:16:56] <jinxer-wm>	 FIRING: SystemdUnitDown: The service unit opentofu-infra-diff.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[03:22:19] <jinxer-wm>	 RESOLVED: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling
[03:50:20] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
[03:59:39] <wikibugs>	 (03update) 10raymond-ndibe: d/changelog: bump to 0.3.7 [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/50
[03:59:59] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
[04:01:12] <wikibugs>	 (03update) 10raymond-ndibe: d/changelog: bump to 0.3.7 [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/50
[04:02:08] <wikibugs>	 (03update) 10raymond-ndibe: d/changelog: bump to 0.3.7 [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/50
[04:06:06] <wikibugs>	 (03update) 10raymond-ndibe: d/changelog: bump to 0.0.14 [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/57
[04:06:28] <wikibugs>	 (03update) 10raymond-ndibe: d/changelog: bump to 0.0.14 [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/57
[04:08:17] <wikibugs>	 (03update) 10raymond-ndibe: d/changelog: bump to 0.0.23 [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/117
[04:08:56] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[04:09:08] <wikibugs>	 (03update) 10raymond-ndibe: d/changelog: bump to 0.0.23 [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/117
[04:11:17] <wikibugs>	 (03update) 10raymond-ndibe: [tests] account for warning messages printed to stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/883 (https://phabricator.wikimedia.org/T400390)
[04:23:25] <wikibugs>	 (03update) 10raymond-ndibe: [deployment] add config to deployment [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/112 (https://phabricator.wikimedia.org/T400064)
[04:23:47] <wikibugs>	 (03update) 10raymond-ndibe: [deployment] add config to deployment [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/112 (https://phabricator.wikimedia.org/T400064)
[04:24:48] <wikibugs>	 (03approved) 10raymond-ndibe: functional_tests: run only webservice tests when it is deployed [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/908 (owner: 10dcaro)
[04:35:14] <wikibugs>	 (03update) 10raymond-ndibe: api: allow protocol to be specified for ports [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/186 (owner: 10dcaro)
[04:48:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[04:53:32] <wikibugs>	 (03update) 10raymond-ndibe: [deployment] add config to deployment [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/112 (https://phabricator.wikimedia.org/T400064)
[05:11:03] <jinxer-wm>	 FIRING: [3x] PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol2005-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange
[05:11:56] <jinxer-wm>	 FIRING: SystemdUnitDown: The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[05:38:48] <wikibugs>	 (03update) 10raymond-ndibe: [deployment] add config to deployment [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/112 (https://phabricator.wikimedia.org/T400064)
[05:54:55] <wikibugs>	 (03update) 10raymond-ndibe: [toolforge,tool_handlers] replace destination_name comparisons with image_name [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/115 (https://phabricator.wikimedia.org/T395076)
[07:22:01] <wikibugs>	 06cloud-services-team, 10Toolforge: puppet broken on tools-elastic-* - https://phabricator.wikimedia.org/T401278 (10taavi) 03NEW p:05Triage→03High
[07:22:03] <wikibugs>	 06cloud-services-team, 10Toolforge: puppet broken on tools-elastic-* - https://phabricator.wikimedia.org/T401278#11063432 (10taavi)
[07:41:16] <wikibugs>	 10Tool-bulkuserinfo: Download Output from API requests as CSV file - https://phabricator.wikimedia.org/T401281 (10Athulvis) 03NEW
[07:43:34] <wikibugs>	 10Tool-bulkuserinfo: Cleaning up and modifying frontend - https://phabricator.wikimedia.org/T401282 (10Athulvis) 03NEW
[07:44:03] <wikibugs>	 10Tool-bulkuserinfo: Cleaning up and modifying frontend - https://phabricator.wikimedia.org/T401282#11063584 (10Athulvis)
[07:44:06] <wikibugs>	 10Tool-bulkuserinfo: Cleaning up and modifying frontend - https://phabricator.wikimedia.org/T401282#11063585 (10Athulvis)
[07:50:35] <wikibugs>	 10Tool-bulkuserinfo: Download Output from API requests as CSV file - https://phabricator.wikimedia.org/T401281#11063606 (10Athulvis)
[07:51:05] <wikibugs>	 10Tool-bulkuserinfo, 03Wikimania-Hackathon-2025: Download Output from API requests as CSV file - https://phabricator.wikimedia.org/T401281#11063607 (10Athulvis)
[08:00:57] <wikibugs>	 06cloud-services-team, 10Toolforge, 13Patch-For-Review: puppet broken on tools-elastic-* - https://phabricator.wikimedia.org/T401278#11063625 (10taavi) 05Open→03Resolved
[08:11:32] <wikibugs>	 (03merge) 10dcaro: functional_tests: run only webservice tests when it is deployed [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/908
[08:26:49] <wikibugs>	 10Tool-bulkuserinfo, 03Wikimania-Hackathon-2025: Bulk User info Fetcher: Download Output from API requests as CSV file - https://phabricator.wikimedia.org/T401281#11063793 (10Athulvis)
[08:40:58] <wmcs-alerts>	 RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance tools-elastic-5 on project tools   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[08:41:25] <wikibugs>	 (03update) 10taavi: Draft: loki_logs: Support following logs [repos/cloud/toolforge/jobs-api] (taavi/fastapi) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/193 (https://phabricator.wikimedia.org/T400916)
[08:41:26] <wikibugs>	 (03update) 10taavi: Draft: Replace Flask with FastAPI [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/192 (https://phabricator.wikimedia.org/T401113)
[08:44:28] <wikibugs>	 (03PS1) 10Gopavasanth: Enhanced tools layout and design [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1176193 (https://phabricator.wikimedia.org/T401275)
[09:02:19] <wikibugs>	 06cloud-services-team, 06Data-Persistence: Decide how to use the new clouddb hosts (clouddb102[2-5]) - https://phabricator.wikimedia.org/T401295 (10fnegri) 03NEW
[09:03:38] <wikibugs>	 06cloud-services-team, 10Data-Services, 06Data-Persistence: Decide how to use the new clouddb hosts (clouddb102[2-5]) - https://phabricator.wikimedia.org/T401295#11063914 (10taavi)
[09:11:03] <jinxer-wm>	 FIRING: [3x] PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol2005-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange
[09:12:11] <jinxer-wm>	 FIRING: SystemdUnitDown: The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[09:34:47] <wikibugs>	 06cloud-services-team, 10Toolforge: Investigate daily disconnections of IRC bots hosted in Toolforge - https://phabricator.wikimedia.org/T400223#11064066 (10fgiunchedi) Disclaimer: I am new to the team and poking around out of curiosity, I may be off base ! This is my current understanding of the issue:  * clo...
[09:55:03] <wikibugs>	 (03update) 10taavi: Draft: loki_logs: Support following logs [repos/cloud/toolforge/jobs-api] (taavi/fastapi) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/193 (https://phabricator.wikimedia.org/T400916)
[09:55:05] <wikibugs>	 (03update) 10taavi: Draft: Replace Flask with FastAPI [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/192 (https://phabricator.wikimedia.org/T401113)
[09:56:42] <wikibugs>	 (03update) 10fnegri: Relicense under Apache 2.0 [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/2
[09:57:50] <wikibugs>	 (03approved) 10taavi: Relicense under Apache 2.0 [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/2 (owner: 10fnegri)
[09:57:53] <wikibugs>	 (03update) 10fnegri: Relicense under Apache 2.0 [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/2
[09:58:26] <wikibugs>	 06cloud-services-team: Onboard Filippo as SRE in Cloud Services - https://phabricator.wikimedia.org/T401091#11064199 (10fgiunchedi) 05Open→03Resolved All done!
[09:59:45] <wikibugs>	 (03update) 10fnegri: Relicense under Apache 2.0 [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/2
[10:09:53] <wikibugs>	 (03update) 10fnegri: Relicense under Apache 2.0 [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/2
[10:12:51] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Use cloud-private network and cfssl certs for instance live migrations - https://phabricator.wikimedia.org/T355145#11064235 (10fgiunchedi)
[10:14:02] <wikibugs>	 (03update) 10taavi: Draft: loki_logs: Support following logs [repos/cloud/toolforge/jobs-api] (taavi/fastapi) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/193 (https://phabricator.wikimedia.org/T400916)
[10:14:04] <wikibugs>	 (03update) 10taavi: Draft: Replace Flask with FastAPI [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/192 (https://phabricator.wikimedia.org/T401113)
[10:14:46] <wikibugs>	 (03open) 10dcaro: global: first commit [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T127367)
[10:14:56] <wikibugs>	 (03update) 10fnegri: Relicense under Apache 2.0 [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/2
[10:17:00] <wikibugs>	 (03update) 10dcaro: global: first commit [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T127367)
[10:18:37] <wikibugs>	 (03merge) 10fnegri: Relicense under Apache 2.0 [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/2
[10:20:48] <wikibugs>	 (03update) 10fnegri: Create .deb package [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/1 (https://phabricator.wikimedia.org/T395266)
[10:22:16] <wikibugs>	 (03update) 10fnegri: Create .deb package [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/1 (https://phabricator.wikimedia.org/T395266)
[10:36:52] <wikibugs>	 (03update) 10taavi: Draft: loki_logs: Support following logs [repos/cloud/toolforge/jobs-api] (taavi/fastapi) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/193 (https://phabricator.wikimedia.org/T400916)
[10:36:53] <wikibugs>	 (03update) 10taavi: Draft: Replace Flask with FastAPI [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/192 (https://phabricator.wikimedia.org/T401113)
[10:40:01] <wikibugs>	 10Tool-centralnotice-banner-editor, 03Wikimania-Hackathon-2025: CentralNotice userscript for sub-subnational areas - https://phabricator.wikimedia.org/T401303 (10Pharos) 03NEW
[10:40:36] <wikibugs>	 10Tool-centralnotice-banner-editor, 03Wikimania-Hackathon-2025: CentralNotice userscript for sub-subnational areas - https://phabricator.wikimedia.org/T401303#11064410 (10Pharos)
[10:41:31] <wikibugs>	 (03update) 10taavi: Draft: Replace Flask with FastAPI [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/192 (https://phabricator.wikimedia.org/T401113)
[10:46:58] <wikibugs>	 06cloud-services-team, 10Toolforge, 10ISA: Request to transfer isa-tool GitHub repository to toolforge organization - https://phabricator.wikimedia.org/T401304 (10Dactylantha) 03NEW
[10:48:57] <wikibugs>	 10Tool-centralnotice-banner-editor, 03Wikimania-Hackathon-2025: CentralNotice userscript for sub-subnational areas - https://phabricator.wikimedia.org/T401303#11064442 (10Novem_Linguae)
[10:50:08] <wikibugs>	 06cloud-services-team, 10Toolforge, 10ISA, 03Wikimania-Hackathon-2025: Request to transfer isa-tool GitHub repository to toolforge organization - https://phabricator.wikimedia.org/T401304#11064446 (10Dactylantha)
[11:15:22] <wikibugs>	 10Tool-centralnotice-banner-editor, 03Wikimania-Hackathon-2025: CentralNotice userscript for sub-subnational areas - https://phabricator.wikimedia.org/T401303#11064508 (10Novem_Linguae) Can you link me to some example code for east/west? I want to double check what the code says for longitude.
[11:21:03] <wikibugs>	 10Tool-centralnotice-banner-editor, 03Wikimania-Hackathon-2025: CentralNotice userscript for sub-subnational areas - https://phabricator.wikimedia.org/T401303#11064518 (10Pharos) It's in the Pennsylvania part of the New York City metro area example, which specifices the northernand eastern part of the state:...
[11:22:15] <wikibugs>	 10Tool-centralnotice-banner-editor, 03Wikimania-Hackathon-2025: CentralNotice userscript for sub-subnational areas - https://phabricator.wikimedia.org/T401303#11064519 (10Novem_Linguae) Hah. I'm glad I double checked with you. I was gonna guess `long`, which would have been incorrect :)
[11:33:05] <wikibugs>	 (03update) 10taavi: Replace Flask with FastAPI [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/192 (https://phabricator.wikimedia.org/T401113)
[11:33:07] <wikibugs>	 (03update) 10taavi: Replace Flask with FastAPI [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/192 (https://phabricator.wikimedia.org/T401113)
[11:41:27] <wikibugs>	 10Tool-centralnotice-banner-editor, 03Wikimania-Hackathon-2025: CentralNotice userscript for sub-subnational areas - https://phabricator.wikimedia.org/T401303#11064549 (10Novem_Linguae) Alright, MVP (minimum viable product) is complete. You can install it by adding the below code to https://meta.wikimedia.org/...
[11:53:43] <wikibugs>	 (03PS2) 10Gopavasanth: Enhanced tools layout and design [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1176193 (https://phabricator.wikimedia.org/T401275)
[12:30:46] <wikibugs>	 (03update) 10dcaro: global: first commit [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T127367)
[12:41:20] <wikibugs>	 (03update) 10dcaro: [tests] account for warning messages printed to stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/883 (https://phabricator.wikimedia.org/T400390) (owner: 10raymond-ndibe)
[12:42:05] <wikibugs>	 10Tool-centralnotice-banner-editor, 03Wikimania-Hackathon-2025: CentralNotice userscript for sub-subnational areas - https://phabricator.wikimedia.org/T401303#11064941 (10Pharos) Beautiful!  Here are some feature requests:  (1) There should be up to four directional statements per subnational unit, enough to d...
[12:43:45] <wikibugs>	 (03update) 10dcaro: [tests] account for warning messages printed to stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/883 (https://phabricator.wikimedia.org/T400390) (owner: 10raymond-ndibe)
[12:44:33] <wikibugs>	 (03update) 10dcaro: [tests] account for warning messages printed to stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/883 (https://phabricator.wikimedia.org/T400390) (owner: 10raymond-ndibe)
[12:45:13] <wikibugs>	 10Tool-centralnotice-banner-editor, 10MediaWiki-extensions-CentralNotice, 03Wikimania-Hackathon-2025: CentralNotice userscript for sub-subnational areas - https://phabricator.wikimedia.org/T401303#11064956 (10Pharos)
[12:46:35] <wikibugs>	 (03update) 10dcaro: [tests] account for warning messages printed to stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/883 (https://phabricator.wikimedia.org/T400390) (owner: 10raymond-ndibe)
[12:48:06] <wikibugs>	 (03update) 10dcaro: [tests] account for warning messages printed to stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/883 (https://phabricator.wikimedia.org/T400390) (owner: 10raymond-ndibe)
[12:49:25] <wikibugs>	 (03update) 10dcaro: [tests] account for warning messages printed to stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/883 (https://phabricator.wikimedia.org/T400390) (owner: 10raymond-ndibe)
[12:50:21] <wikibugs>	 (03update) 10dcaro: [tests] account for warning messages printed to stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/883 (https://phabricator.wikimedia.org/T400390) (owner: 10raymond-ndibe)
[12:55:42] <wikibugs>	 (03update) 10dcaro: global: first commit [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T127367)
[12:57:40] <wikibugs>	 (03update) 10taavi: Draft: loki_logs: Support following logs [repos/cloud/toolforge/jobs-api] (taavi/fastapi) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/193 (https://phabricator.wikimedia.org/T400916)
[12:57:42] <wikibugs>	 (03update) 10taavi: Replace Flask with FastAPI [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/192 (https://phabricator.wikimedia.org/T401113)
[13:00:15] <wikibugs>	 (03update) 10dcaro: global: first commit [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T127367)
[13:01:13] <wikibugs>	 (03update) 10dcaro: global: first commit [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T127367)
[13:02:37] <wikibugs>	 (03approved) 10dcaro: utils: Provide asyncio version of peek() [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/84 (owner: 10taavi)
[13:03:14] <wikibugs>	 (03update) 10taavi: utils: Provide asyncio version of peek() [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/84
[13:04:46] <wikibugs>	 (03CR) 10Ssingh: [C:03+1] Add more Traffic repositories [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1175950 (owner: 10BCornwall)
[13:10:47] <wikibugs>	 (03update) 10dcaro: [tests] account for warning messages printed to stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/883 (https://phabricator.wikimedia.org/T400390) (owner: 10raymond-ndibe)
[13:11:03] <jinxer-wm>	 FIRING: [3x] PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol2005-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange
[13:12:11] <jinxer-wm>	 FIRING: SystemdUnitDown: The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[13:18:04] <wikibugs>	 (03update) 10dcaro: [tests] account for warning messages printed to stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/883 (https://phabricator.wikimedia.org/T400390) (owner: 10raymond-ndibe)
[13:19:11] <wikibugs>	 06cloud-services-team, 10Toolforge: 500 Internal Server Error when trying to access ssh keys on toolsadmin - https://phabricator.wikimedia.org/T401318 (10Soni) 03NEW
[13:23:23] <wikibugs>	 06cloud-services-team, 10Striker: 500 Internal Server Error when trying to access ssh keys on toolsadmin - https://phabricator.wikimedia.org/T401318#11065201 (10taavi) `counterexample Traceback (most recent call last):   File "/opt/lib/poetry/striker-2uZo5AhP-py3.11/lib/python3.11/site-packages/django/core/han...
[13:25:23] <wikibugs>	 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad: hw troubleshooting: disk sdj failure for cloudcephosd1013.eqiad.wmnet - https://phabricator.wikimedia.org/T401319 (10fnegri) 03NEW
[13:26:01] <wikibugs>	 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad: hw troubleshooting: disk sdj failure for cloudcephosd1013.eqiad.wmnet - https://phabricator.wikimedia.org/T401319#11065222 (10fnegri)
[13:26:02] <wikibugs>	 06cloud-services-team: KernelErrors Server cloudcephosd1013 logged kernel errors - https://phabricator.wikimedia.org/T399366#11065223 (10fnegri)
[13:26:49] <wikibugs>	 06cloud-services-team: KernelErrors Server cloudcephosd1013 logged kernel errors - https://phabricator.wikimedia.org/T399366#11065224 (10fnegri) 05Open→03Resolved a:03fnegri Follow-up: I asked DCops to take out the failed drive in {T401319}.
[13:29:41] <wikibugs>	 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T400225#11065255 (10fnegri) 05Open→03Resolved a:03fnegri The alert is not firing anymore so I will resolve this.
[13:32:12] <wikibugs>	 (03update) 10dcaro: [tests] account for warning messages printed to stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/883 (https://phabricator.wikimedia.org/T400390) (owner: 10raymond-ndibe)
[13:32:14] <wikibugs>	 06cloud-services-team: PuppetDisabled Puppet disabled on cloudcontrol2010-dev:9100 - https://phabricator.wikimedia.org/T400357#11065268 (10fnegri) 05Open→03Resolved a:03fnegri
[13:32:26] <wikibugs>	 06cloud-services-team: PuppetDisabled Puppet disabled on cloudcontrol2006-dev:9100 - https://phabricator.wikimedia.org/T400381#11065273 (10fnegri) 05Open→03Resolved a:03fnegri
[13:32:56] <wikibugs>	 06cloud-services-team: PuppetFailure Puppet has failed on cloudbackup1002-dev:9100 - https://phabricator.wikimedia.org/T400650#11065275 (10fnegri) 05Open→03Resolved a:03fnegri
[13:33:18] <wikibugs>	 06cloud-services-team: PuppetDisabled Puppet disabled on cloudcontrol2005-dev:9100 - https://phabricator.wikimedia.org/T400356#11065278 (10fnegri) 05Open→03Resolved a:03fnegri
[13:34:52] <wikibugs>	 06cloud-services-team: SystemdUnitDown The systemd unit backup_cinder_volumes.service on node cloudbackup1002-dev has been failing for more than two hours. - https://phabricator.wikimedia.org/T400655#11065283 (10fnegri) 05Open→03Resolved a:03fnegri
[13:34:53] <wikibugs>	 06cloud-services-team: SystemdUnitDown The systemd unit backup_cinder_volumes.service on node cloudbackup1001-dev has been failing for more than two hours. - https://phabricator.wikimedia.org/T400298#11065285 (10fnegri) 05Open→03Resolved a:03fnegri
[13:39:11] <wikibugs>	 (03update) 10dcaro: [tests] account for warning messages printed to stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/883 (https://phabricator.wikimedia.org/T400390) (owner: 10raymond-ndibe)
[13:39:37] <wikibugs>	 (03approved) 10dcaro: Replace Flask with FastAPI [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/192 (https://phabricator.wikimedia.org/T401113) (owner: 10taavi)
[13:48:45] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
[13:49:21] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
[13:53:22] <wikibugs>	 (03update) 10taavi: loki_logs: Support following logs [repos/cloud/toolforge/jobs-api] (taavi/fastapi) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/193 (https://phabricator.wikimedia.org/T400916)
[13:53:23] <wikibugs>	 (03update) 10taavi: loki_logs: Support following logs [repos/cloud/toolforge/jobs-api] (taavi/fastapi) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/193 (https://phabricator.wikimedia.org/T400916)
[14:04:34] <wikibugs>	 (03update) 10dcaro: [tests] account for warning messages printed to stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/883 (https://phabricator.wikimedia.org/T400390) (owner: 10raymond-ndibe)
[14:09:01] <wikibugs>	 06cloud-services-team, 10Striker: 500 Internal Server Error when trying to access ssh keys on toolsadmin - https://phabricator.wikimedia.org/T401318#11065433 (10Soni) Would deleting the stored ssh key on the Toolforge side solve this?
[14:12:00] <wikibugs>	 06cloud-services-team, 10Striker: 500 Internal Server Error when trying to access ssh keys on toolsadmin - https://phabricator.wikimedia.org/T401318#11065446 (10taavi) Does https://idm.wikimedia.org/keymanagement/ recognize and/or allow you to manage the existing key on your account?
[14:16:47] <wikibugs>	 10Tool-centralnotice-banner-editor, 10MediaWiki-extensions-CentralNotice, 03Wikimania-Hackathon-2025: CentralNotice userscript for sub-subnational areas - https://phabricator.wikimedia.org/T401303#11065479 (10Novem_Linguae) Can you please add some sample code to https://meta.wikimedia.org/wiki/Talk:North_Ame...
[14:18:52] <wikibugs>	 (03update) 10dcaro: [tests] account for warning messages printed to stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/883 (https://phabricator.wikimedia.org/T400390) (owner: 10raymond-ndibe)
[14:24:24] <wikibugs>	 06cloud-services-team, 10Striker: 500 Internal Server Error when trying to access ssh keys on toolsadmin - https://phabricator.wikimedia.org/T401318#11065509 (10dcaro) I think that there's a typo in the key that was uploaded, the one that's in ldap starts like: ` ssh-rsa AAAAB4NzaC1yc2EAAAADAQABAAABAQCaBSmjYcM...
[14:31:31] <wmcs-alerts>	 FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on toolsbeta-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates
[14:31:35] <wikibugs>	 (03CR) 10Ladsgroup: [C:03+2] "It will be deployed in a day, let me know if it doesn't." [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1175950 (owner: 10BCornwall)
[14:31:47] <wikibugs>	 (03update) 10dcaro: [tests] account for warning messages printed to stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/883 (https://phabricator.wikimedia.org/T400390) (owner: 10raymond-ndibe)
[14:43:10] <wikibugs>	 06cloud-services-team, 10Striker: 500 Internal Server Error when trying to access ssh keys on toolsadmin - https://phabricator.wikimedia.org/T401318#11065578 (10Soni) ` ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpzVhg5BZCq4y+Y+N0VLq5xNoiv37E04WFoC8eyvXdAsCBx7p7h6tpUMFNxu/guWBXo7hTNLo2hp51PVT39R/MAxPgGCmdynhgolxvXG...
[14:43:41] <wikibugs>	 (03CR) 10Ladsgroup: [C:03+2] "It'll be deployed in a day, let me know if it doesn't." [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1175948 (https://phabricator.wikimedia.org/T347623) (owner: 10BCornwall)
[14:44:56] <wikibugs>	 (03Merged) 10jenkins-bot: Update Traffic repo locations to GitLab [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1175948 (https://phabricator.wikimedia.org/T347623) (owner: 10BCornwall)
[14:44:59] <wikibugs>	 (03Merged) 10jenkins-bot: Add more Traffic repositories [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1175950 (owner: 10BCornwall)
[14:56:48] <wikibugs>	 (03merge) 10taavi: Replace Flask with FastAPI [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/192 (https://phabricator.wikimedia.org/T401113)
[14:56:50] <wikibugs>	 (03update) 10taavi: loki_logs: Support following logs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/193 (https://phabricator.wikimedia.org/T400916)
[14:59:52] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.394-20250806145659-85863be5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/909 (https://phabricator.wikimedia.org/T401113)
[14:59:57] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.394-20250806145659-85863be5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/909 (https://phabricator.wikimedia.org/T401113)
[15:01:46] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
[15:03:15] <wikibugs>	 06cloud-services-team, 10Toolforge, 10ISA, 03Wikimania-Hackathon-2025: Request to transfer isa-tool GitHub repository to toolforge organization - https://phabricator.wikimedia.org/T401304#11065657 (10bd808) @Dactylantha What is your relationship with the tool? It is not clear to me that you are one of the...
[15:12:02] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
[15:12:20] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
[15:13:06] <wikibugs>	 (03merge) 10taavi: utils: Provide asyncio version of peek() [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/84
[15:14:14] <wikibugs>	 (03open) 10taavi: d/changelog: bump to 1.6.12 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/85
[15:14:17] <wikibugs>	 (03update) 10taavi: d/changelog: bump to 1.6.12 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/85
[15:15:58] <wikibugs>	 06cloud-services-team: SystemdUnitDown The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. - https://phabricator.wikimedia.org/T401161#11065691 (10fnegri)
[15:16:01] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Cloud VPS project creation cookbook times out really often - https://phabricator.wikimedia.org/T398712#11065692 (10fnegri)
[15:16:30] <wikibugs>	 06cloud-services-team: SystemdUnitDown The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. - https://phabricator.wikimedia.org/T401161#11065697 (10fnegri) This was caused by {T398712}.  The `magnum` project was created but `tofu apply` timed out and did...
[15:16:48] <wikibugs>	 (03merge) 10taavi: d/changelog: bump to 1.6.12 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/85
[15:16:53] <wikibugs>	 06cloud-services-team: SystemdUnitDown The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. - https://phabricator.wikimedia.org/T401161#11065702 (10fnegri) 05Open→03Resolved a:03fnegri
[15:16:56] <jinxer-wm>	 RESOLVED: SystemdUnitDown: The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[15:21:14] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
[15:21:30] <wikibugs>	 (03update) 10taavi: loki_logs: Support following logs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/193 (https://phabricator.wikimedia.org/T400916)
[15:21:52] <wikibugs>	 (03merge) 10taavi: jobs-api: bump to 0.0.394-20250806145659-85863be5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/909 (https://phabricator.wikimedia.org/T401113) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[15:22:21] <wikibugs>	 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [jobs-api] Migrate to FastAPI - https://phabricator.wikimedia.org/T401113#11065739 (10taavi) 05Open→03Resolved
[15:25:08] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld
[15:25:10] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld
[15:30:07] <wikibugs>	 (03update) 10dcaro: [tests] account for warning messages printed to stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/883 (https://phabricator.wikimedia.org/T400390) (owner: 10raymond-ndibe)
[15:30:49] <jinxer-wm>	 FIRING: [3x] PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol2005-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange
[15:40:49] <jinxer-wm>	 FIRING: [3x] PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol2005-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange
[15:42:33] <wikibugs>	 06cloud-services-team, 10Toolforge, 10ISA, 03Wikimania-Hackathon-2025: Request to transfer isa-tool GitHub repository to toolforge organization - https://phabricator.wikimedia.org/T401304#11065832 (10bd808) Is the https://github.com/bjhoareAM/isa-tool mirror maintained by Gerrit automation or something els...
[15:51:31] <wmcs-alerts>	 RESOLVED: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on toolsbeta-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates
[15:58:25] <wikibugs>	 (03merge) 10bd808: lighttpd: Use "lighttpd" as webservice type [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/80 (https://phabricator.wikimedia.org/T401014)
[16:00:49] <jinxer-wm>	 RESOLVED: PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol2010-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange
[16:46:29] <wikibugs>	 (03approved) 10dcaro: [tests] account for warning messages printed to stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/883 (https://phabricator.wikimedia.org/T400390) (owner: 10raymond-ndibe)
[16:46:39] <wikibugs>	 (03merge) 10dcaro: [tests] account for warning messages printed to stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/883 (https://phabricator.wikimedia.org/T400390) (owner: 10raymond-ndibe)
[16:47:12] <wikibugs>	 10Toolforge (Toolforge iteration 23), 13Patch-For-Review: [toolforge-deploy.tests] account for warning messages printed to stderr - https://phabricator.wikimedia.org/T400390#11066032 (10dcaro) 05In progress→03Resolved
[17:04:10] <wikibugs>	 (03open) 10bd808: d/changelog: bump to 0.103.17 [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/83
[17:04:14] <wikibugs>	 (03approved) 10dcaro: loki_logs: Support following logs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/193 (https://phabricator.wikimedia.org/T400916) (owner: 10taavi)
[17:04:31] <wikibugs>	 (03update) 10dcaro: global: first commit [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T127367)
[17:05:23] <wikibugs>	 (03update) 10dcaro: global: first commit [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T127367)
[17:07:50] <wikibugs>	 (03update) 10taavi: loki_logs: Support following logs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/193 (https://phabricator.wikimedia.org/T400916)
[17:12:18] <wikibugs>	 (03merge) 10taavi: loki_logs: Support following logs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/193 (https://phabricator.wikimedia.org/T400916)
[17:12:48] <wikibugs>	 (03open) 10dcaro: logs_api: add the option to enable logs-api [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/75
[17:15:49] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.395-20250806171229-63780c9e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/910 (https://phabricator.wikimedia.org/T400916)
[17:16:39] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
[17:20:03] <wmcs-alerts>	 FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-36 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses
[17:21:21] <logmsgbot_cloud>	 !log bd808@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli
[17:21:40] <logmsgbot_cloud>	 !log bd808@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli
[17:26:44] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
[17:26:49] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
[17:33:06] <logmsgbot_cloud>	 !log bd808@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli (T401014)
[17:33:11] <stashbot>	 T401014: php7.3 webservice type unable to run PHP - https://phabricator.wikimedia.org/T401014
[17:34:18] <logmsgbot_cloud>	 !log bd808@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli (T401014)
[17:35:14] <wikibugs>	 06cloud-services-team, 10Toolforge: php7.3 webservice type unable to run PHP - https://phabricator.wikimedia.org/T401014#11066194 (10LucasWerkmeister) I stopped+started the webservice in the lucaswerkmeister-test tool and now https://lucaswerkmeister-test.toolforge.org/hello.php is working \o/ thanks @bd808!
[17:36:43] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
[17:37:04] <wikibugs>	 (03update) 10dcaro: logs_api: add the option to enable logs-api [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/75
[17:38:07] <wikibugs>	 (03merge) 10taavi: jobs-api: bump to 0.0.395-20250806171229-63780c9e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/910 (https://phabricator.wikimedia.org/T400916) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[17:38:44] <wikibugs>	 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [jobs-api] Support following logs from Loki - https://phabricator.wikimedia.org/T400916#11066210 (10taavi) 05Open→03Resolved
[17:38:47] <logmsgbot_cloud>	 !log bd808@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli (T401014)
[17:38:52] <stashbot>	 T401014: php7.3 webservice type unable to run PHP - https://phabricator.wikimedia.org/T401014
[17:39:05] <wikibugs>	 (03update) 10dcaro: global: first commit [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T127367)
[17:39:05] <logmsgbot_cloud>	 !log bd808@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli (T401014)
[17:39:52] <wm-bot2>	 Change on 12wikitech.wikimedia.org a page Help:Toolforge/Running jobs was modified, changed by Taavi-WMF link https://wikitech.wikimedia.org/w/index.php?diff=2330880 edit summary: /* Internal log storage */ no longer the case
[17:41:19] <logmsgbot_cloud>	 !log bd808@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli (T401014)
[17:41:34] <logmsgbot_cloud>	 !log bd808@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli (T401014)
[17:43:06] <wikibugs>	 06cloud-services-team, 10Toolforge: What is the preferred method of exporting metrics from jobs? - https://phabricator.wikimedia.org/T401235#11066253 (10taavi) →14Duplicate dup:03T366923
[17:43:07] <wikibugs>	 06cloud-services-team, 10Toolforge: Set up new Prometheus instance for user-created data - https://phabricator.wikimedia.org/T366923#11066255 (10taavi)
[17:43:16] <wikibugs>	 06cloud-services-team, 10Toolforge: What is the preferred method of exporting metrics from jobs? - https://phabricator.wikimedia.org/T401235#11066259 (10taavi) →14Duplicate dup:03T362012
[17:43:22] <wikibugs>	 06cloud-services-team, 10Toolforge: [jobs] Allow configuration of Promethus scraping of a specific endpoint for publication in grafana.wmcloud.org - https://phabricator.wikimedia.org/T362012#11066261 (10taavi)
[17:44:28] <wikibugs>	 06cloud-services-team, 10Toolforge, 10ISA, 03Wikimania-Hackathon-2025: Request to transfer isa-tool GitHub repository to toolforge organization - https://phabricator.wikimedia.org/T401304#11066265 (10taavi) Is there a particular reason this needs to move to a proprietary platform instead of using any of th...
[17:45:05] <wikibugs>	 06cloud-services-team, 10Toolforge: `toolforge jobs logs` returns nothing if started too early. - https://phabricator.wikimedia.org/T401073#11066272 (10taavi) 05Open→03Resolved a:03taavi I believe {T400916} fixed this.
[17:49:40] <wikibugs>	 (03PS1) 10David Caro: deploy: skip buster bastion when deploying webservice [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1176291
[17:53:30] <logmsgbot_cloud>	 !log bd808@cloudcumin1001 tools START - Cookbook wmcs.toolforge.run_tests
[17:53:50] <wikibugs>	 (03CR) 10CI reject: [V:04-1] deploy: skip buster bastion when deploying webservice [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1176291 (owner: 10David Caro)
[17:54:23] <wikibugs>	 (03update) 10dcaro: global: first commit [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T127367)
[17:54:32] <logmsgbot_cloud>	 !log bd808@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0)
[17:58:06] <wikibugs>	 (03PS2) 10David Caro: deploy: skip buster bastion when deploying webservice [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1176291
[17:58:49] <wikibugs>	 (03approved) 10bd808: d/changelog: bump to 0.103.17 [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/83
[17:58:56] <wikibugs>	 (03merge) 10bd808: d/changelog: bump to 0.103.17 [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/83
[18:06:53] <wikibugs>	 (03update) 10vriaa: Basic banner implementation [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/1
[18:11:25] <wikibugs>	 06cloud-services-team, 10Toolforge: php7.3 webservice type unable to run PHP - https://phabricator.wikimedia.org/T401014#11066327 (10bd808) 05Open→03Resolved p:05Triage→03Medium a:03bd808 `lang=shell-session $ become bd808-test2 tools.bd808-test2@tools-bastion-12:~$ webservice php7.2 restart DEPR...
[18:24:59] <wikibugs>	 06cloud-services-team, 10Toolforge: Trove for cluebotng-review? - https://phabricator.wikimedia.org/T401347 (10DamianZaremba) 03NEW
[18:29:08] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Project-requests): Trove for cluebotng-review? - https://phabricator.wikimedia.org/T401347#11066386 (10taavi)
[18:40:28] <wmcs-alerts>	 FIRING: PuppetAgentFailure: Puppet agent failure detected on instance metricsinfra-alertmanager-2 in project metricsinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure
[18:55:28] <wmcs-alerts>	 RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance metricsinfra-alertmanager-2 in project metricsinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure
[20:35:04] <wikibugs>	 (03CR) 10BryanDavis: [C:03+1] deploy: skip buster bastion when deploying webservice [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1176291 (owner: 10David Caro)
[20:57:47] <wikibugs>	 (03open) 10bd808: Updates from Bryan's use of the deploy process [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/84
[20:57:55] <wikibugs>	 (03update) 10bd808: Updates from Bryan's use of the deploy process [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/84
[20:57:59] <wikibugs>	 (03update) 10bd808: Updates from Bryan's use of the deploy process [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/84
[21:25:07] <wikibugs>	 10Tool-centralnotice-banner-editor, 03Wikimania-Hackathon-2025: CentralNotice userscript for sub-subnational areas - https://phabricator.wikimedia.org/T401303#11066732 (10Novem_Linguae) a:03Novem_Linguae `#3` is done since it was easy. `#1` and `#2` are harder so haven't done them yet.  I'm removing all tags...
[21:26:42] <wikibugs>	 (03update) 10bd808: lighttpd: Use "lighttpd" as webservice type [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/80 (https://phabricator.wikimedia.org/T401014)
[22:21:01] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: [tofu-cloudvps] cloudvps_puppet_prefix.hiera settings show dirty diffs based on YAML canonicalization - https://phabricator.wikimedia.org/T398643#11066842 (10bd808) >>! In T398643#11015358, @bd808 wrote: > https://developer.hashicorp.com/terraform/plugin/sdkv2/best-practices...
[23:02:18] <wikibugs>	 06cloud-services-team, 10Toolforge, 13Patch-For-Review: puppet broken on tools-elastic-* - https://phabricator.wikimedia.org/T401278#11066927 (10colewhite) Sorry about the noise!  It's an optional parameter now.
[23:55:56] <jinxer-wm>	 FIRING: SystemdUnitDown: The service unit kiwix-mirror-update.service is in failed status on host clouddumps1001. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[23:57:40] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: [tofu-cloudvps] cloudvps_puppet_prefix.hiera settings show dirty diffs based on YAML canonicalization - https://phabricator.wikimedia.org/T398643#11067045 (10bd808) I spent time over the last two days poking at this which mostly means I spent time figuring out how to compile...