[00:35:38] <wikibugs>	 10VPS-project-Codesearch: Allow searching for plain text in CodeSearch - https://phabricator.wikimedia.org/T381325#12185691 (10Ladsgroup) >>! In T381325#11943202, @SomeRandomDeveloper wrote: > https://gerrit.wikimedia.org/r/c/labs/codesearch/+/1267343 will allow implementing this option.  I merged that patch, th...
[01:11:30] <wikibugs>	 (03CR) 10MusikAnimal: "It looks like the queries are still using the underscored variant. `/ec-timecard/enwiki/Example?format=wikitext` errors out for me." [labs/xtools] - 10https://gerrit.wikimedia.org/r/1320920 (https://phabricator.wikimedia.org/T433394) (owner: 10Alien4444)
[01:36:50] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.roll_reboot_osds (T429387)
[01:36:51] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.roll_reboot_osds (exit_code=99) (T429387)
[01:36:56] <stashbot>	 T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387
[01:37:06] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.roll_reboot_osds (T429387)
[01:45:56] <wikibugs>	 (03CR) 10MusikAnimal: [C:03+2] "Works in my testing, thank you!" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1320944 (https://phabricator.wikimedia.org/T344090) (owner: 10Alien4444)
[01:47:00] <wikibugs>	 (03Merged) 10jenkins-bot: Repository: use tables not schemata in getDbList [labs/xtools] - 10https://gerrit.wikimedia.org/r/1320944 (https://phabricator.wikimedia.org/T344090) (owner: 10Alien4444)
[01:52:42] <wikibugs>	 10Tool-wikinewsie: Headlines failing to post on Wikinewsie - https://phabricator.wikimedia.org/T433796#12185726 (10Pharos) 05Open→03Resolved a:03Pharos
[02:06:13] <wikibugs>	 (03CR) 10MusikAnimal: SimpleEditCounter: add latest edit and log (031 comment) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1276371 (https://phabricator.wikimedia.org/T198872) (owner: 10Alien4444)
[02:17:03] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.roll_reboot_osds (exit_code=0) (T429387)
[02:17:09] <stashbot>	 T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387
[02:17:15] <wikibugs>	 10Cloud-VPS (Quota-requests), 06Moderator-Tools-Team, 10WikiWho: Remove increased quota for project wikiwho - https://phabricator.wikimedia.org/T434023#12185768 (10Andrew) +1 Thanks for the follow-up!
[02:24:38] <wikibugs>	 (03CR) 10MusikAnimal: "It's about 300ms slower for my enwiki account. That's enough I think to at least make it optional. Add to the growing list of checkboxes. " [labs/xtools] - 10https://gerrit.wikimedia.org/r/1276066 (https://phabricator.wikimedia.org/T276520) (owner: 10Alien4444)
[02:28:47] <jinxer-wm>	 FIRING: NodeDown: Node cloudcephosd1044 has been down for long. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephosd1044 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown
[02:49:14] <wikibugs>	 (03CR) 10MusikAnimal: [C:03+2] "Doesn't seem to have a dramatic effect on speeds, so we can give it a try. Note however Pages Created is one of the tools that times out a" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1275974 (https://phabricator.wikimedia.org/T352737) (owner: 10Alien4444)
[02:50:10] <wikibugs>	 (03Merged) 10jenkins-bot: PagesCreated: add mimetype column for ns6 [labs/xtools] - 10https://gerrit.wikimedia.org/r/1275974 (https://phabricator.wikimedia.org/T352737) (owner: 10Alien4444)
[03:26:32] <wikibugs>	 10Tool-itwiki, 10MediaWiki-Action-API, 06MediaWiki-API-Platform-Team: Missing categories from generator=categorymembers (just 1 instead of 2) in itwiki - https://phabricator.wikimedia.org/T433922#12185851 (10KineticPelagic) As part of my Clinic Duty rotation, I am moving this task to Bugs & Chores, modeling...
[04:58:12] <wikibugs>	 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [jobs-api] support exposing continuous jobs to the internet - https://phabricator.wikimedia.org/T423408#12185875 (10Raymond_Ndibe) >>! In T423408#12177977, @dcaro wrote: > Forgot to add the changelog  updated the documentation to add `-...
[04:58:43] <wikibugs>	 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [jobs-api] support exposing continuous jobs to the internet - https://phabricator.wikimedia.org/T423408#12185877 (10Raymond_Ndibe) 05In progress→03Resolved
[04:58:49] <wikibugs>	 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [jobs-cli] support publishing continuous job to the internet - https://phabricator.wikimedia.org/T423410#12185881 (10Raymond_Ndibe) 05In progress→03Resolved
[05:18:16] <wikibugs>	 (03open) 10raymond-ndibe: models: support publish option [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/180 (https://phabricator.wikimedia.org/T433355)
[05:18:19] <wikibugs>	 (03update) 10raymond-ndibe: models: support publish option [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/180 (https://phabricator.wikimedia.org/T433355)
[05:19:31] <wikibugs>	 (03update) 10raymond-ndibe: models: support publish option [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/180 (https://phabricator.wikimedia.org/T433355)
[05:51:44] <wikibugs>	 (03open) 10raymond-ndibe: components-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1367 (https://phabricator.wikimedia.org/T433355)
[05:53:31] <wikibugs>	 (03update) 10raymond-ndibe: components-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1367 (https://phabricator.wikimedia.org/T433355)
[05:55:14] <wikibugs>	 (03update) 10raymond-ndibe: components-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1367 (https://phabricator.wikimedia.org/T433355)
[06:07:25] <wikibugs>	 (03update) 10raymond-ndibe: models: support publish option [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/180 (https://phabricator.wikimedia.org/T433355)
[06:56:40] <wikibugs>	 (03update) 10dcaro: tests: refactors, improvements and testing some untested branches [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/362 (owner: 10raymond-ndibe)
[06:57:47] <wikibugs>	 10Data-Services, 10Toolforge, 06tools-platform-team: The gb_block_email column of the centralauth_p.globalblocks view is missing in the toolforge replicas - https://phabricator.wikimedia.org/T434044 (10Count_Count) 03NEW
[07:10:28] <wikibugs>	 10Data-Services, 06tools-platform-team: The gb_block_email column of the centralauth_p.globalblocks view is missing in the toolforge replicas - https://phabricator.wikimedia.org/T434044#12186006 (10JJMC89)
[07:11:07] <wikibugs>	 10Data-Services, 06tools-platform-team: The gb_block_email column of the centralauth_p.globalblocks view is missing in the wiki replicas - https://phabricator.wikimedia.org/T434044#12186008 (10JJMC89)
[07:24:39] <wikibugs>	 (03update) 10dcaro: refactor: move observability fields out of CommonJob into ObservabilityJob [repos/cloud/toolforge/jobs-api] (move_cmd_out_of_common_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/363 (owner: 10raymond-ndibe)
[07:28:36] <wikibugs>	 10Data-Services, 06tools-platform-team: The gb_block_email column of the centralauth_p.globalblocks view is missing in the wiki replicas - https://phabricator.wikimedia.org/T434044#12186054 (10JJMC89) The view is working as [[ https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/5465c99538fd13e43...
[07:28:58] <wikibugs>	 (03update) 10dcaro: models: move cmd field out of CommonJob into job-type models [repos/cloud/toolforge/jobs-api] (increase_test_coverage) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/364 (owner: 10raymond-ndibe)
[07:29:06] <wikibugs>	 10Data-Services, 06tools-platform-team: Expose the gb_block_email column of the centralauth_p.globalblocks table in the wiki replicas view - https://phabricator.wikimedia.org/T434044#12186056 (10JJMC89)
[07:35:43] <wikibugs>	 (03update) 10dcaro: models: move cmd field out of CommonJob into job-type models [repos/cloud/toolforge/jobs-api] (increase_test_coverage) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/364 (owner: 10raymond-ndibe)
[07:44:17] <wikibugs>	 (03update) 10dcaro: models: support publish option [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/180 (https://phabricator.wikimedia.org/T433355) (owner: 10raymond-ndibe)
[07:45:46] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12186087 (10Marostegui) 05Open→03Resolved All new hosts are now serving traffic. Closing this task and will open a new one to decommi...
[07:45:48] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA: Set up replication on new hosts clouddb102[2-5] - https://phabricator.wikimedia.org/T408692#12186091 (10Marostegui)
[07:47:17] <wikibugs>	 10Data-Services, 06tools-platform-team: Expose the gb_block_email column of the centralauth_p.globalblocks table in the wiki replicas view - https://phabricator.wikimedia.org/T434044#12186092 (10Count_Count) @JJMC89, sorry but a column missing in the Toolforge replica view for no good reason is a bug not a mis...
[07:48:58] <wikibugs>	 10Data-Services, 06tools-platform-team, 06DBA: Decommission clouddb1013-clouddb1020 - https://phabricator.wikimedia.org/T434048 (10Marostegui) 03NEW
[07:49:06] <wikibugs>	 (03open) 10dcaro: d/changelog: bump to 16.2.1 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/172 (https://phabricator.wikimedia.org/T399080 https://phabricator.wikimedia.org/T433405)
[07:49:29] <wikibugs>	 10Data-Services, 06tools-platform-team, 06DBA: Decommission clouddb1013-clouddb1020 - https://phabricator.wikimedia.org/T434048#12186114 (10Marostegui)
[07:49:31] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12186115 (10Marostegui)
[07:49:42] <wikibugs>	 10Data-Services, 06tools-platform-team, 06DBA: Decommission clouddb1013-clouddb1020 - https://phabricator.wikimedia.org/T434048#12186116 (10Marostegui) p:05Triage→03Medium a:03Marostegui
[07:59:02] <wikibugs>	 (03open) 10dcaro: bump_version: copied latest version from builds-cli [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/173
[08:00:27] <wikibugs>	 (03update) 10dcaro: bump_version: copied latest version from builds-cli [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/173
[08:00:36] <wikibugs>	 (03update) 10dcaro: bump_version: move to trixie + update script [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/173
[08:00:37] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Cloud VPS Debian Bullseye deprecation - https://phabricator.wikimedia.org/T401804#12186159 (10valerio.bozzolan) Hi. Can I ask if the project `wlm-it-visual` is OK? I ask because I've received a notification from the kind Komla Sapaty, but in Hor...
[08:02:23] <wikibugs>	 (03update) 10dcaro: bump_version: move to trixie + update script [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/173
[08:03:09] <wikibugs>	 10Data-Services, 06tools-platform-team, 06DBA: Decommission clouddb1013-clouddb1020 - https://phabricator.wikimedia.org/T434048#12186165 (10Marostegui) All those clouddb hosts have been depooled. Only the new ones are serving traffic now fyi @taavi @fnegri
[08:03:32] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Cloud VPS Debian Bullseye deprecation - https://phabricator.wikimedia.org/T401804#12186167 (10valerio.bozzolan) Ah, ignore my last comment thanks :) As `wlm-it-visual` was deleted: https://phabricator.wikimedia.org/T429723#12172741
[08:08:36] <wikibugs>	 06tools-platform-team: [clis] use trixie to bump versions - https://phabricator.wikimedia.org/T434050 (10dcaro) 03NEW
[08:14:50] <wikibugs>	 (03update) 10dcaro: bump_version: move to trixie + update script [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/173
[08:15:43] <wikibugs>	 (03update) 10dcaro: bump_version: move to trixie + update script [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/173
[08:15:49] <wikibugs>	 (03update) 10dcaro: bump_version: move to trixie + update script [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/173
[08:16:21] <wikibugs>	 (03update) 10dcaro: bump_version: move to trixie + update script [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/173
[08:17:16] <wikibugs>	 10Toolforge, 06tools-platform-team: [clis] use trixie to bump versions - https://phabricator.wikimedia.org/T434050#12186206 (10taavi)
[08:19:23] <wikibugs>	 (03close) 10dcaro: d/changelog: bump to 16.2.1 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/172 (https://phabricator.wikimedia.org/T399080 https://phabricator.wikimedia.org/T433405)
[08:24:46] <wikibugs>	 10Toolforge, 06tools-platform-team: [clis] use trixie to bump versions - https://phabricator.wikimedia.org/T434050#12186246 (10dcaro)
[08:29:42] <wikibugs>	 (03open) 10dcaro: bump_version: move to trixie [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/145
[08:36:23] <wikibugs>	 (03update) 10dcaro: bump_version: move to trixie [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/145
[08:37:28] <wikibugs>	 (03open) 10dcaro: bump_version: move to trixie [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/116
[08:39:23] <wikibugs>	 (03update) 10dcaro: bump_version: move to trixie [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/116
[08:40:50] <wikibugs>	 10Toolforge, 06tools-platform-team: [clis] use trixie to bump versions - https://phabricator.wikimedia.org/T434050#12186305 (10dcaro)
[08:47:07] <wikibugs>	 (03update) 10dcaro: bump_version: move to trixie [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/145
[08:49:54] <wikibugs>	 (03update) 10dcaro: bump_version: move to trixie [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/145
[08:51:12] <wikibugs>	 (03open) 10dcaro: bump version trixie [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/71
[08:52:02] <wikibugs>	 (03update) 10dcaro: bump version trixie [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/71
[08:54:10] <wikibugs>	 (03update) 10dcaro: bump version trixie [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/71
[08:54:31] <wikibugs>	 (03update) 10dcaro: bump version trixie [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/71
[08:55:13] <wikibugs>	 (03update) 10dcaro: bump_version: move to trixie [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/71
[09:01:44] <wikibugs>	 (03update) 10dcaro: bump_version: move to trixie [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/116
[09:03:59] <wikibugs>	 (03update) 10dcaro: bump_version: move to trixie [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/71
[09:04:27] <wikibugs>	 (03update) 10dcaro: bump_version: move to trixie [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/145
[09:05:30] <wikibugs>	 (03update) 10dcaro: bump_version: move to trixie + update script [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/173
[09:05:42] <wikibugs>	 (03update) 10dcaro: bump_version: move to trixie [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/71
[09:06:36] <wikibugs>	 10Toolforge, 06tools-platform-team: [clis] use trixie to bump versions - https://phabricator.wikimedia.org/T434050#12186370 (10dcaro)
[09:09:43] <wikibugs>	 (03open) 10dcaro: bump_version: move to trixie [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/118
[09:10:28] <wikibugs>	 (03update) 10dcaro: bump_version: move to trixie [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/118
[09:10:54] <wikibugs>	 10Toolforge, 06tools-platform-team: [clis] use trixie to bump versions - https://phabricator.wikimedia.org/T434050#12186374 (10dcaro)
[09:12:40] <wikibugs>	 (03open) 10dcaro: bump_version: move to trixie [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/88
[09:14:04] <wikibugs>	 (03update) 10dcaro: bump_version: move to trixie [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/88
[09:14:29] <wikibugs>	 10Toolforge, 06tools-platform-team: [clis] use trixie to bump versions - https://phabricator.wikimedia.org/T434050#12186379 (10dcaro)
[09:29:43] <wikibugs>	 10Tool-itwiki, 10MediaWiki-Action-API, 06MediaWiki-API-Platform-Team: Missing categories from generator=categorymembers (just 1 instead of 2) in itwiki - https://phabricator.wikimedia.org/T433922#12186413 (10Zabe) a:03Zabe
[09:30:52] <wikibugs>	 (03open) 10dcaro: bump_version: move to trixie [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/20
[09:42:32] <wikibugs>	 10Tool-ldap: Show SSH key fingerprints - https://phabricator.wikimedia.org/T434054 (10taavi) 03NEW
[09:49:59] <wikibugs>	 10Toolforge, 06tools-platform-team, 07OKR-Work: [loki,alloy,logs-api] reconfigure the labels and add metadata - https://phabricator.wikimedia.org/T432969#12186517 (10dcaro) >>! In T432969#12183954, @TLepage-WMF wrote: > - [ ] Deploy the Alloy changes to production.  Keep in mind that alloy takes a long time...
[09:50:27] <wikibugs>	 (03update) 10dcaro: bump_version: move to trixie [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/20
[10:13:46] <wmcs-alerts>	 FIRING: ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_api_svc_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[10:15:14] <wmcs-alerts>	 FIRING: [3x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-8.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown
[10:18:46] <wmcs-alerts>	 FIRING: [6x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[10:20:14] <wmcs-alerts>	 RESOLVED: [5x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-8.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown
[10:23:46] <wmcs-alerts>	 FIRING: [7x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[10:24:01] <wmcs-alerts>	 FIRING: [7x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[10:28:46] <wmcs-alerts>	 RESOLVED: [5x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[10:30:56] <wmcs-alerts>	 FIRING: ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[10:35:56] <wmcs-alerts>	 RESOLVED: [5x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[10:40:56] <wmcs-alerts>	 FIRING: [8x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[10:45:56] <wmcs-alerts>	 RESOLVED: [5x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[10:48:14] <wmcs-alerts>	 FIRING: [3x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-gateway-2.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown
[10:52:03] <wmcs-alerts>	 FIRING: [6x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[10:53:14] <wmcs-alerts>	 RESOLVED: [8x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-7.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown
[10:54:14] <wmcs-alerts>	 FIRING: [6x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-7.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown
[10:55:17] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Wikimedia-Developer-Portal: Update Dev Portal staging to Debian Bookworm - https://phabricator.wikimedia.org/T429490#12186723 (10KBach) The new VPS instance (devportal-demo02) is now up and responding to requests at https://developer-portal.wmcloud.org.  I copied th...
[10:57:03] <wmcs-alerts>	 FIRING: [5x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[10:57:42] <wikibugs>	 (03PS1) 10Ladsgroup: frontend: Add literal option [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1321532 (https://phabricator.wikimedia.org/T381325)
[10:59:14] <wmcs-alerts>	 RESOLVED: [6x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-7.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown
[11:00:34] <wikibugs>	 10VPS-project-Codesearch, 13Patch-For-Review: Allow searching for plain text in CodeSearch - https://phabricator.wikimedia.org/T381325#12186756 (10Ladsgroup) This works but hound acts a bit weird on the API side. For example  https://codesearch-backend.wmcloud.org/core/api/v1/search?q=%24wgDBName&literal=1
[11:02:03] <wmcs-alerts>	 FIRING: [6x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[11:07:03] <wmcs-alerts>	 RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[11:08:56] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[11:13:56] <wmcs-alerts>	 FIRING: [7x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[11:16:14] <wmcs-alerts>	 FIRING: [3x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-9.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown
[11:18:56] <wmcs-alerts>	 FIRING: [8x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[11:21:14] <wmcs-alerts>	 FIRING: [12x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-7.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown
[11:21:18] <wikibugs>	 06Toolforge-standards-committee, 10Tools, 07Privacy: views.toolforge.org loads third-party resources - https://phabricator.wikimedia.org/T434065 (10taavi) 03NEW
[11:23:56] <wmcs-alerts>	 FIRING: [8x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[11:26:14] <wmcs-alerts>	 RESOLVED: [11x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-control-7.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown
[11:28:56] <wmcs-alerts>	 RESOLVED: [6x] ProbeDown: Service tools-k8s-haproxy-8:30004 has failed probes (http_infra_tracing_loki_svc_tools_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[12:10:08] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Wikimedia-Developer-Portal: Update Dev Portal staging to Debian Bookworm - https://phabricator.wikimedia.org/T429490#12186906 (10KBach) 05Open→03In progress
[13:01:54] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Wikimedia-Developer-Portal: Update Dev Portal staging to Debian Bookworm - https://phabricator.wikimedia.org/T429490#12187076 (10KBach) I realized that the instance should use Bookworm instead of Trixie so I created a new one (devportal-demo03) and deleted the previ...
[13:03:41] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Wikimedia-Developer-Portal: Update Dev Portal staging to Debian Bookworm - https://phabricator.wikimedia.org/T429490#12187083 (10taavi) > I realized that the instance should use Bookworm instead of Trixie  Why?
[13:09:13] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Wikimedia-Developer-Portal: Update Dev Portal staging to Debian Bookworm - https://phabricator.wikimedia.org/T429490#12187102 (10KBach) >>! In T429490#12187083, @taavi wrote: >> I realized that the instance should use Bookworm instead of Trixie >  > Why?  Because th...
[13:18:47] <wikibugs>	 (03update) 10dcaro: bump_version: move to trixie [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/20
[13:21:05] <wikibugs>	 (03update) 10dcaro: bump_version: move to trixie [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/118
[13:21:24] <wikibugs>	 (03update) 10dcaro: bump_version: move to trixie [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/116
[13:21:36] <wikibugs>	 (03update) 10dcaro: bump_version: move to trixie + update script [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/173
[13:22:01] <wikibugs>	 (03update) 10dcaro: bump_version: move to trixie [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/71
[13:22:19] <wikibugs>	 (03update) 10dcaro: bump_version: move to trixie [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/88
[13:22:31] <wikibugs>	 (03update) 10dcaro: bump_version: move to trixie [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/20
[13:23:17] <wikibugs>	 10Toolforge, 06tools-platform-team: [clis] use trixie to bump versions - https://phabricator.wikimedia.org/T434050#12187123 (10dcaro)
[13:28:54] <wikibugs>	 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Network unavailable on a few VMs - https://phabricator.wikimedia.org/T432426#12187143 (10Andrew) Trixie host f78be803-6d05-43cc-9b40-a8748ff56204 statanalyser.statanalyser.eqiad1.wikimedia.cloud just fell off the network.  This does not seem to be...
[13:30:45] <wikibugs>	 10Toolforge, 06tools-platform-team: [clis] drop the build scripts in favor of gitlab-ci-local - https://phabricator.wikimedia.org/T434071#12187169 (10dcaro)
[13:31:07] <wikibugs>	 10Toolforge, 06tools-platform-team: [clis] drop the build scripts in favor of gitlab-ci-local - https://phabricator.wikimedia.org/T434071#12187172 (10dcaro)
[13:31:18] <wikibugs>	 10Toolforge, 06tools-platform-team: [clis] use trixie to bump versions - https://phabricator.wikimedia.org/T434050#12187174 (10dcaro) p:05Triage→03Medium
[13:32:05] <wikibugs>	 10Toolforge, 06tools-platform-team: [all] first of the month dependency upgrades - https://phabricator.wikimedia.org/T433878#12187177 (10dcaro)
[13:33:19] <wikibugs>	 (03CR) 10Majavah: [C:03+1] Add copy_certmanager_images cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1320996 (https://phabricator.wikimedia.org/T408786) (owner: 10Andrew Bogott)
[13:34:39] <wikibugs>	 10Toolforge, 06tools-platform-team, 07OKR-Work: [api-gateway] Specify allowed urls for superusers - https://phabricator.wikimedia.org/T433930#12187206 (10dcaro)
[13:35:42] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation): Debian Bullseye deprecation extension for statanalyser - https://phabricator.wikimedia.org/T433570#12187209 (10Andrew) Hello @Leaderboard -- I see that this VM did not survive the in-place upgrade and is now off the network. If you need assistance with data rescue or VM...
[13:47:43] <wikibugs>	 (03open) 10dcaro: start-devenv: don't use recursive on files [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/336
[13:47:54] <wikibugs>	 10Toolforge, 06tools-platform-team: [lima-kilo] shows error when copying home dirs - https://phabricator.wikimedia.org/T433849#12187272 (10dcaro) p:05Triage→03Low a:03dcaro
[13:49:56] <wikibugs>	 (03update) 10dcaro: start-devenv: don't use recursive on files [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/336
[13:50:43] <wikibugs>	 06cloud-services-team, 06tools-platform-team, 07Documentation: Update list of upstream dependencies on mw:Upstream_projects - https://phabricator.wikimedia.org/T328345#12187290 (10aputhin) 05Open→03In progress
[13:51:54] <wikibugs>	 (03approved) 10dcaro: remove "wait: true" from certmanager ansible script [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/335 (https://phabricator.wikimedia.org/T408786) (owner: 10andrew)
[13:52:19] <wikibugs>	 (03update) 10dcaro: remove "wait: true" from certmanager ansible script [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/335 (https://phabricator.wikimedia.org/T408786) (owner: 10andrew)
[14:01:59] <wikibugs>	 10Tool-clip2commons: Long videos fail during encoding because output is buffered in memory - https://phabricator.wikimedia.org/T434083 (10Dev) 03NEW
[14:02:26] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component cert-manager (T408786)
[14:02:31] <stashbot>	 T408786: Upgrade cert-manager past 1.15 - https://phabricator.wikimedia.org/T408786
[14:03:33] <wikibugs>	 (03update) 10dcaro: start-devenv: don't use recursive on files [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/336
[14:06:03] <wikibugs>	 10Tool-clip2commons: Remember previously selected license - https://phabricator.wikimedia.org/T434084 (10Dev) 03NEW
[14:09:45] <wikibugs>	 10Data-Services, 06tools-platform-team: Expose the gb_block_email column of the centralauth_p.globalblocks table in the wiki replicas view - https://phabricator.wikimedia.org/T434044#12187470 (10aputhin) p:05Triage→03Medium
[14:11:43] <wikibugs>	 10Tool-clip2commons: iOS WebKit form resets under memory pressure - https://phabricator.wikimedia.org/T434086 (10Dev) 03NEW
[14:14:02] <wikibugs>	 10Tool-clip2commons: Automatically add uploaded video to Wikidata (P10) - https://phabricator.wikimedia.org/T434087 (10Dev) 03NEW
[14:16:24] <wikibugs>	 10Tool-clip2commons: Support additional structured data statements - https://phabricator.wikimedia.org/T434088 (10Dev) 03NEW
[14:19:16] <wikibugs>	 10Tool-clip2commons: Add a Vitest unit test suite - https://phabricator.wikimedia.org/T434090 (10Dev) 03NEW
[14:19:45] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component cert-manager (T408786)
[14:19:50] <stashbot>	 T408786: Upgrade cert-manager past 1.15 - https://phabricator.wikimedia.org/T408786
[14:23:17] <wikibugs>	 10Tool-clip2commons: Cross-browser capability diagnostics page - https://phabricator.wikimedia.org/T434091 (10Dev) 03NEW
[14:23:34] <wikibugs>	 10Striker, 06tools-platform-team, 10Toolhub, 07Documentation: Expose default documentation space in Wikitech for Toolforge tools in ToolsAdmin and/or Toolhub - https://phabricator.wikimedia.org/T433325#12187569 (10aputhin) p:05Triage→03Low
[14:24:55] <wikibugs>	 10Tool-clip2commons: Complete remaining translations - https://phabricator.wikimedia.org/T434092 (10Dev) 03NEW
[14:25:00] <wikibugs>	 10Tool-wikiloves: WL Sciences : even years are missing - https://phabricator.wikimedia.org/T379626#12187591 (10Jony) 05Open→03Resolved a:03Jony Added
[14:27:44] <wikibugs>	 10Striker, 06tools-platform-team, 07Documentation: Expose default documentation space in Wikitech for Toolforge tools in ToolsAdmin and/or Toolhub - https://phabricator.wikimedia.org/T433325#12187616 (10taavi) The toolinfo field has a field for an URL to documentation. I think it would be pretty simple to cr...
[14:28:06] <wikibugs>	 06tools-platform-team: Establish design principles for parity between interfaces (CLI vs REST API) - https://phabricator.wikimedia.org/T434093 (10aputhin) 03NEW
[14:28:52] <wikibugs>	 10Toolforge, 06tools-platform-team: [logs-cli] Add `--direction` flag to choose the direction of the logs shown - https://phabricator.wikimedia.org/T433869#12187630 (10aputhin) p:05Triage→03Low
[14:29:09] <wikibugs>	 10Toolforge, 06tools-platform-team: Establish design principles for parity between interfaces (CLI vs REST API) - https://phabricator.wikimedia.org/T434093#12187643 (10taavi)
[14:30:37] <wikibugs>	 10Tool-clip2commons: Add multilingual MediaInfo captions during upload - https://phabricator.wikimedia.org/T434095 (10Dev) 03NEW
[14:42:01] <wikibugs>	 10Toolforge, 06tools-platform-team: [jobs-cli,envvars-cli,toolforge-cli,builds-cli] enable usage stats collection - https://phabricator.wikimedia.org/T433954#12187728 (10aputhin) We need a better notion of sizing and a preliminary design of how this would look like.  Considerations: - Users need to be informed...
[14:46:57] <wikibugs>	 10Toolforge, 06tools-platform-team: [logs-cli] add a `--first` flag, or replace `--latest` with `--lines` - https://phabricator.wikimedia.org/T433870#12187746 (10aputhin) Prioritization depends on T434093 and possibly T433954
[14:47:18] <wikibugs>	 10Toolforge, 06tools-platform-team: [logs-cli] Add `--direction` flag to choose the direction of the logs shown - https://phabricator.wikimedia.org/T433869#12187752 (10aputhin) Prioritization depends on T434093 and possibly T433954
[15:03:50] <wikibugs>	 10Toolforge, 06tools-platform-team: [jobs-api] Allow storing one-off jobs - https://phabricator.wikimedia.org/T433868#12187832 (10aputhin) We need more input on how one-of jobs are currently used, starting with getting CLI usage stats (T433954), and also in parallel reaching out to some users to understand the...
[15:24:50] <wikibugs>	 10Toolforge, 06tools-platform-team: [logs-cli] Add `--direction` flag to choose the direction of the logs shown - https://phabricator.wikimedia.org/T433869#12187898 (10dcaro)
[15:26:57] <wikibugs>	 10Toolforge, 06tools-platform-team: Establish design principles for parity between interfaces (CLI vs REST API) - https://phabricator.wikimedia.org/T434093#12187907 (10dcaro) Some previous work on this can be found scattered in docs and such, I'll be adding them when I find them, firs one: https://docs.google....
[15:29:06] <wikibugs>	 10Toolforge, 06tools-platform-team: [logs-cli] add a `--first` flag, or replace `--latest` with `--lines` - https://phabricator.wikimedia.org/T433870#12187915 (10dcaro)
[15:31:16] <wikibugs>	 10Toolforge, 06tools-platform-team: Establish design principles for parity between interfaces (CLI vs REST API) - https://phabricator.wikimedia.org/T434093#12187933 (10dcaro) More related notes here https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/Decision_record_T348749_T...
[15:41:59] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Shutdown timeline extension requests for maintained projects - https://phabricator.wikimedia.org/T434103 (10bd808) 03NEW
[15:42:54] <wikibugs>	 10Toolforge, 06tools-platform-team: [jobs-cli,envvars-cli,toolforge-cli,builds-cli] enable usage stats collection - https://phabricator.wikimedia.org/T433954#12188029 (10dcaro) >>! In T433954#12187728, @aputhin wrote: > We need a better notion of sizing and a preliminary design of how this would look like. >...
[15:44:49] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation): Extension request for account-creation-assistance - https://phabricator.wikimedia.org/T433749#12188032 (10JJMC89)
[15:44:50] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation): Debian Bullseye deprecation extension for statanalyser - https://phabricator.wikimedia.org/T433570#12188033 (10JJMC89)
[15:44:51] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Shutdown timeline extension requests for maintained projects - https://phabricator.wikimedia.org/T434103#12188031 (10JJMC89)
[16:00:53] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Shutdown timeline extension requests for maintained projects - https://phabricator.wikimedia.org/T434103#12188132 (10bd808)
[16:02:24] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Cloud VPS Debian Bullseye deprecation - https://phabricator.wikimedia.org/T401804#12188138 (10taavi) p:05Unbreak!→03Triage
[16:04:15] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06SRE, 13Patch-For-Review: Modernise memcached systemd unit / sync, and make it presentable - https://phabricator.wikimedia.org/T273950#12188151 (10jijiki)
[16:06:59] <wikibugs>	 10Tool-clip2commons: Add multilingual MediaInfo captions during upload - https://phabricator.wikimedia.org/T434095#12188169 (10Anushka10patel) a:03Anushka10patel Hello! I am a new contributor and I would like to work on this task.  I am interested in contributing to Clip2Commons and would like to take this up...
[16:07:59] <wikibugs>	 10Tool-clip2commons: Complete remaining translations - https://phabricator.wikimedia.org/T434092#12188174 (10Anushka10patel) a:03Anushka10patel Hello! I am a new contributor and I would like to work on this task.  I am interested in contributing by completing the remaining interface translations for the suppor...
[16:09:00] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06SRE, 13Patch-For-Review: Modernise memcached systemd unit / sync, and make it presentable - https://phabricator.wikimedia.org/T273950#12188180 (10jijiki)
[16:10:35] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-platform-team: Send follow up messages to  Cloud VPS project maintainers about Debian Bullseye deprecation - https://phabricator.wikimedia.org/T433334#12188188 (10aputhin) 05In progress→03Resolved
[16:11:07] <wikibugs>	 10Tool-clip2commons: Remember previously selected license - https://phabricator.wikimedia.org/T434084#12188192 (10Anushka10patel) a:03Anushka10patel Hello! I am a new contributor and I would like to work on this task.  I am interested in contributing by completing the remaining interface translations for the s...
[16:35:23] <wikibugs>	 06cloud-services-team, 10Toolforge, 06tools-infrastructure-team: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.33 - https://phabricator.wikimedia.org/T408785#12188372 (10Andrew) This is deployed on toolsbeta and seems to be working. I will deploy on tools proper tomorrow.
[16:37:57] <wikibugs>	 (03update) 10dcaro: toolforge_deploy: allow following the logs after deploy [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/333
[16:38:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[16:38:43] <wikibugs>	 06cloud-services-team, 10Toolforge, 06tools-infrastructure-team, 13Patch-For-Review: Upgrade cert-manager past 1.15 - https://phabricator.wikimedia.org/T408786#12188406 (10Andrew) 1.20.3 is now deployed on toolsbeta and seems to be working. I will deploy on tools proper tomorrow.
[16:40:48] <wikibugs>	 (03CR) 10Andrew Bogott: [C:03+2] Add copy_certmanager_images cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1320996 (https://phabricator.wikimedia.org/T408786) (owner: 10Andrew Bogott)
[16:41:23] <wikibugs>	 (03approved) 10andrew: toolforge_deploy: allow following the logs after deploy [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/333 (owner: 10dcaro)
[16:41:43] <wikibugs>	 (03merge) 10dcaro: toolforge_deploy: allow following the logs after deploy [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/333
[16:44:22] <wikibugs>	 (03Merged) 10jenkins-bot: Add copy_certmanager_images cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1320996 (https://phabricator.wikimedia.org/T408786) (owner: 10Andrew Bogott)
[16:48:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[16:50:53] <wikibugs>	 10Tool-inteGraality: Support reference columns in property dashboards - https://phabricator.wikimedia.org/T428637#12188489 (10JeanFred) A lot of the checks that we planned is now implemented, per https://www.wikidata.org/wiki/Wikidata:InteGraality/Concepts#Reference_columns : * Any reference  `S*`  The statement...
[17:09:06] <wikibugs>	 10Toolforge, 06tools-platform-team, 07OKR-Work: [loki,alloy,logs-api] reconfigure the labels and add metadata - https://phabricator.wikimedia.org/T432969#12188596 (10TLepage-WMF) Update: I am reconsidering the idea of doing the Alloy schema change all at once along with two logs-api deployments, because that...
[17:14:20] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Shutdown timeline extension requests for maintained projects - https://phabricator.wikimedia.org/T434103#12188643 (10bd808)
[17:15:57] <wikibugs>	 (03update) 10renovatebot: Update python Docker tag to v3.14.6 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/18
[17:15:57] <wikibugs>	 (03update) 10renovatebot: Update python Docker tag to v3.14.7 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/18
[17:16:27] <wikibugs>	 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-api,loki] Implement a system-level logging endpoint (write) - https://phabricator.wikimedia.org/T432565#12188674 (10dcaro)
[17:16:28] <wikibugs>	 10Toolforge, 06tools-platform-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, 07OKR-Work: [jobs-api] add certificate to authenticate against logs-api - https://phabricator.wikimedia.org/T434120 (10dcaro) 03NEW p:05Triage→03High
[17:17:52] <wikibugs>	 (03open) 10dcaro: deployment: added certificate with client rights [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/365
[17:21:12] <wikibugs>	 10Toolforge, 06tools-platform-team, 07OKR-Work: [loki,alloy,logs-api] reconfigure the labels and add metadata - https://phabricator.wikimedia.org/T432969#12188722 (10dcaro) > I don’t think ingesting logs with the additional (static) label `source` is going to significantly affect storage/indexing on Loki?  S...
[17:52:23] <wikibugs>	 10Cloud-VPS, 06tools-infrastructure-team: Install and run dhcpd on VMs - https://phabricator.wikimedia.org/T432782#12188884 (10Andrew) 05Open→03Invalid
[17:52:45] <wikibugs>	 06cloud-services-team, 10VPS-Projects: Dumps: replace Bullseye nfs server with Trixie - https://phabricator.wikimedia.org/T430590#12188886 (10Andrew) 05Open→03Resolved
[17:53:57] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation): Migrate WMCS-managed NFS servers off of Bullseye - https://phabricator.wikimedia.org/T401812#12188888 (10Andrew) 05Open→03Resolved let's try this again :)
[17:56:00] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: Standardize on opentofu management of all projects in the default keystone domain - https://phabricator.wikimedia.org/T410659#12188896 (10Andrew) a:05Andrew→03None I think this is correct but I'm not working on it.
[18:00:30] <wikibugs>	 06tools-infrastructure-team: Review capacity projection setup for toolforge and cloud-vps - https://phabricator.wikimedia.org/T434128 (10Andrew) 03NEW
[18:01:41] <wikibugs>	 06tools-infrastructure-team: Review capacity projection setup for toolforge and cloud-vps - https://phabricator.wikimedia.org/T434128#12188917 (10Andrew)
[18:11:47] <wikibugs>	 06tools-infrastructure-team: Review capacity projection setup for toolforge and cloud-vps - https://phabricator.wikimedia.org/T434128#12188946 (10Andrew) The top row of [[ https://grafana.wikimedia.org/d/000000579/wmcs-openstack-eqiad-summary | this dashboard  ]] provides a good overview of how full our metal in...
[18:16:02] <wikibugs>	 (03update) 10renovatebot: Update python Docker tag to v3.14.7 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/18
[18:17:24] <wikibugs>	 06tools-infrastructure-team: cloud-vps: review our CPU overprovisioning policy - https://phabricator.wikimedia.org/T434131 (10Andrew) 03NEW
[18:50:38] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation): Debian Bullseye deprecation extension for statanalyser - https://phabricator.wikimedia.org/T433570#12189102 (10Leaderboard) In-place upgrade @Andrew? Sorry, why was the VM upgraded without asking me?
[18:57:10] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Cloud VPS Debian Bullseye deprecation - https://phabricator.wikimedia.org/T401804#12189127 (10Leaderboard) Given that an in-place upgrade was apparently performed for my VM (T433570), I'm now pretty confused. Is it that VMs that are not already...
[19:05:03] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Shutdown timeline extension requests for maintained projects - https://phabricator.wikimedia.org/T434103#12189172 (10Cyberpower678)
[19:33:55] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation): Debian Bullseye deprecation extension for statanalyser - https://phabricator.wikimedia.org/T433570#12189276 (10Andrew) Hello @Leaderboard!  I haven't done anything on that VM other than investigate after it started reporting as down this morning.  The VM was built Jan....
[19:36:00] <wikibugs>	 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Network unavailable on a few VMs - https://phabricator.wikimedia.org/T432426#12189291 (10Andrew) >>! In T432426#12187143, @Andrew wrote: > Trixie host f78be803-6d05-43cc-9b40-a8748ff56204 statanalyser.statanalyser.eqiad1.wikimedia.cloud just fell...
[19:53:35] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,magnum
[19:54:15] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,magnum
[20:15:12] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Cloud VPS Debian Bullseye deprecation - https://phabricator.wikimedia.org/T401804#12189402 (10bd808) >>! In T401804#12189127, @Leaderboard wrote: > Given that an in-place upgrade was apparently performed for my VM (T433570), I'm now pretty confu...
[21:56:01] <wikibugs>	 (03PS3) 10Alien4444: EditCounter: fix local-time breaking timecard [labs/xtools] - 10https://gerrit.wikimedia.org/r/1320920 (https://phabricator.wikimedia.org/T433394)
[21:57:05] <wikibugs>	 (03CR) 10Alien4444: "Sorry, hadn't paid enough attention there." [labs/xtools] - 10https://gerrit.wikimedia.org/r/1320920 (https://phabricator.wikimedia.org/T433394) (owner: 10Alien4444)
[22:02:45] <wikibugs>	 (03PS6) 10Alien4444: SimpleEditCounter: add latest edit and log [labs/xtools] - 10https://gerrit.wikimedia.org/r/1276371 (https://phabricator.wikimedia.org/T198872)
[22:02:45] <wikibugs>	 (03CR) 10Alien4444: SimpleEditCounter: add latest edit and log (031 comment) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1276371 (https://phabricator.wikimedia.org/T198872) (owner: 10Alien4444)
[22:18:10] <wikibugs>	 (03CR) 10MusikAnimal: SimpleEditCounter: add latest edit and log (031 comment) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1276371 (https://phabricator.wikimedia.org/T198872) (owner: 10Alien4444)
[22:23:35] <wikibugs>	 (03CR) 10MusikAnimal: "`/ec-timecard/enwiki/MusikPuppet` gives:" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1320920 (https://phabricator.wikimedia.org/T433394) (owner: 10Alien4444)
[23:08:15] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 10Tool-spacemedia, 06tools-infrastructure-team, and 2 others: Cloud Services shared IP (static NAT for external communications) often rate limited by YouTube for video downloads - https://phabricator.wikimedia.org/T236446#12189840 (10Lionel_Scheepmans) In the frame of my...