[00:36:58] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] move custom validations out of api models [repos/cloud/toolforge/jobs-api] (refactor_imagename_field_in_models) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/150 (https://phabricator.wikimedia.org/T389118)
[00:40:24] <wikibugs>	 10wikitech.wikimedia.org, 06MediaWiki-Platform-Team, 05SUL3: Fix stuck old cookies on Wikitech - https://phabricator.wikimedia.org/T389433#10655184 (10Tgr)
[03:09:18] <wikibugs>	 (03PS1) 10Chuckonwumelu: Add new profile [labs/private] - 10https://gerrit.wikimedia.org/r/1129595
[05:41:40] <wikibugs>	 (03CR) 10Arnaudb: [C:03+1] vrts: add profile::vrts::exim_deny_senders with fake value [labs/private] - 10https://gerrit.wikimedia.org/r/1129374 (https://phabricator.wikimedia.org/T389079) (owner: 10Dzahn)
[08:03:07] <wikibugs>	 06cloud-services-team, 10Data-Services: Update views on global_block_whitelist tables - https://phabricator.wikimedia.org/T387663#10655949 (10ops-monitoring-bot) Cookbook cookbooks.sre.wikireplicas.update-views run by taavi: Started updating wiki replica views
[08:09:07] <wikibugs>	 06cloud-services-team, 10Data-Services: Update views on global_block_whitelist tables - https://phabricator.wikimedia.org/T387663#10655969 (10ops-monitoring-bot) Cookbook cookbooks.sre.wikireplicas.update-views started by taavi completed: - an-redacteddb1001.eqiad.wmnet (**PASS**)   - Ran Puppet agent   - Ran...
[08:09:21] <wikibugs>	 06cloud-services-team, 10Data-Services: Update views on global_block_whitelist tables - https://phabricator.wikimedia.org/T387663#10655972 (10taavi) 05Open→03Resolved
[08:47:44] <wikibugs>	 10Toolforge (Toolforge iteration 19), 13Patch-For-Review: [jobs-api] Split the API, business, and k8s models - https://phabricator.wikimedia.org/T359808#10656079 (10taavi) This doesn't seem to have had any progress since last May - why is it bouncing through every iteration as "in progress"?
[09:11:55] <wikibugs>	 06cloud-services-team, 13Patch-For-Review: ProbeDown - https://phabricator.wikimedia.org/T388379#10656159 (10taavi) 05Open→03Resolved a:03cmooney
[09:12:22] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: ProbeDown - https://phabricator.wikimedia.org/T388379#10656163 (10taavi)
[09:13:00] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge, 05Cloud-Services-Origin-Alert, 07Cloud-Services-Worktype-Unplanned: 2025-03-15 Tools NFS hiccup - https://phabricator.wikimedia.org/T388965#10656172 (10taavi) Anything left to do here?
[09:22:22] <wikibugs>	 10wikitech.wikimedia.org: ☂  Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10656196 (10Ladsgroup) This is not really related to this task. Please file a separate task with idbm (bitu) so the right people could investigate it. It has nothing to do with SUL <-> Wi...
[09:41:42] <jinxer-wm>	 FIRING: [2x] AlertLintProblem: Linting problems found for NeutronAgentDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem
[09:41:51] <wikibugs>	 06cloud-services-team: AlertLintProblem - https://phabricator.wikimedia.org/T389467 (10phaultfinder) 03NEW
[09:59:07] <wikibugs>	 06cloud-services-team: AlertLintProblem - https://phabricator.wikimedia.org/T389467#10656269 (10aborrero) →14Duplicate dup:03T373878
[09:59:08] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: openstack: fix missing prometheus metrics - https://phabricator.wikimedia.org/T373878#10656271 (10aborrero)
[10:05:57] <wikibugs>	 (03PS5) 10Arturo Borrero Gonzalez: wmcs.openstack.tofu: when dealing with MR checkouts, operate on a different dir [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022)
[10:06:31] <wikibugs>	 (03PS6) 10Arturo Borrero Gonzalez: wmcs.openstack.tofu: when dealing with MR checkouts, operate on a different dir [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022)
[10:08:27] <wikibugs>	 (03PS7) 10Arturo Borrero Gonzalez: wmcs.openstack.tofu: when dealing with MR checkouts, operate on a different dir [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022)
[10:11:44] <wikibugs>	 (03CR) 10CI reject: [V:04-1] wmcs.openstack.tofu: when dealing with MR checkouts, operate on a different dir [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022) (owner: 10Arturo Borrero Gonzalez)
[10:12:42] <wikibugs>	 (03PS8) 10Arturo Borrero Gonzalez: wmcs.openstack.tofu: when dealing with MR checkouts, operate on a different dir [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022)
[10:16:44] <wikibugs>	 (03CR) 10CI reject: [V:04-1] wmcs.openstack.tofu: when dealing with MR checkouts, operate on a different dir [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022) (owner: 10Arturo Borrero Gonzalez)
[10:27:48] <wikibugs>	 (03PS9) 10Arturo Borrero Gonzalez: wmcs.openstack.tofu: when dealing with MR checkouts, operate on a different dir [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022)
[10:35:57] <wikibugs>	 (03CR) 10CI reject: [V:04-1] wmcs.openstack.tofu: when dealing with MR checkouts, operate on a different dir [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022) (owner: 10Arturo Borrero Gonzalez)
[10:39:11] <wikibugs>	 (03PS10) 10Arturo Borrero Gonzalez: wmcs.openstack.tofu: when dealing with MR checkouts, operate on a different dir [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022)
[10:42:52] <wikibugs>	 (03CR) 10CI reject: [V:04-1] wmcs.openstack.tofu: when dealing with MR checkouts, operate on a different dir [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022) (owner: 10Arturo Borrero Gonzalez)
[10:47:53] <wikibugs>	 (03PS11) 10Arturo Borrero Gonzalez: wmcs.openstack.tofu: when dealing with MR checkouts, operate on a different dir [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022)
[10:51:12] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10Wikidata, and 2 others: Wikibase selenium tests timeout, seemingly due to "memory compaction" events on CI VMs - https://phabricator.wikimedia.org/T281122#10656478 (10zeljkofilipin)
[10:51:29] <wikibugs>	 (03CR) 10CI reject: [V:04-1] wmcs.openstack.tofu: when dealing with MR checkouts, operate on a different dir [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022) (owner: 10Arturo Borrero Gonzalez)
[11:03:38] <wikibugs>	 (03PS12) 10Arturo Borrero Gonzalez: wmcs.openstack.tofu: when dealing with MR checkouts, operate on a different dir [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022)
[11:04:46] <wikibugs>	 (03PS13) 10Arturo Borrero Gonzalez: wmcs.openstack.tofu: when dealing with MR checkouts, operate on a different dir [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022)
[11:08:30] <wikibugs>	 (03CR) 10CI reject: [V:04-1] wmcs.openstack.tofu: when dealing with MR checkouts, operate on a different dir [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022) (owner: 10Arturo Borrero Gonzalez)
[12:21:24] <wikibugs>	 (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/11
[12:37:43] <wikibugs>	 (03PS14) 10Arturo Borrero Gonzalez: wmcs.openstack.tofu: when dealing with MR checkouts, operate on a different dir [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022)
[12:44:24] <wikibugs>	 (03CR) 10CI reject: [V:04-1] wmcs.openstack.tofu: when dealing with MR checkouts, operate on a different dir [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022) (owner: 10Arturo Borrero Gonzalez)
[13:55:12] <wikibugs>	 (03CR) 10Arturo Borrero Gonzalez: Add new profile (031 comment) [labs/private] - 10https://gerrit.wikimedia.org/r/1129595 (owner: 10Chuckonwumelu)
[14:04:18] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.vps.add_user_to_project for user 'chuckonwumelu' in role 'member'
[14:04:24] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'chuckonwumelu' in role 'member'
[14:04:30] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.vps.add_user_to_project for user 'chuckonwumelu' in role 'member'
[14:04:36] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'chuckonwumelu' in role 'member'
[14:04:41] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 paws START - Cookbook wmcs.vps.add_user_to_project for user 'chuckonwumelu' in role 'member'
[14:04:46] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 paws END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'chuckonwumelu' in role 'member'
[14:05:51] <wikibugs>	 06cloud-services-team: Onboard Chuck Onwumelu - https://phabricator.wikimedia.org/T386715#10657192 (10aborrero)
[14:14:28] <wikibugs>	 (03PS15) 10Arturo Borrero Gonzalez: wmcs.openstack.tofu: when dealing with MR checkouts, operate on a different dir [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022)
[14:18:45] <wikibugs>	 (03CR) 10CI reject: [V:04-1] wmcs.openstack.tofu: when dealing with MR checkouts, operate on a different dir [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022) (owner: 10Arturo Borrero Gonzalez)
[14:20:29] <wikibugs>	 10Tools, 07Privacy: wmtran Tool Loads External Resource - https://phabricator.wikimedia.org/T333894#10657230 (10Aklapper) > This is my tool, why wasn't I assigned to this task immediately? Because assigning someone else often gives the wrong impression that people work on something while they don't. It's a bad...
[14:28:33] <wikibugs>	 06cloud-services-team: Onboard Chuck Onwumelu - https://phabricator.wikimedia.org/T386715#10657252 (10Chuckonwumelu)
[15:09:23] <wikibugs>	 10Cloud Services Proposals, 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services, 06Data-Persistence, 10Data-Platform-SRE (2025.03.01 - 2025.03.21): Decision request - Who runs wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607#10657381 (10aborrero) 05In progress→03Stalled Stall...
[15:32:04] <wikibugs>	 (03PS16) 10Arturo Borrero Gonzalez: wmcs.openstack.tofu: when dealing with MR checkouts, operate on a different dir [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022)
[15:44:42] <wikibugs>	 10Horizon: HTTP 500 trying to load interfaces, UI just keeps spinning - https://phabricator.wikimedia.org/T362138#10657568 (10dancy) Same as T380531
[15:58:46] <wikibugs>	 (03CR) 10Arturo Borrero Gonzalez: "recheck" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022) (owner: 10Arturo Borrero Gonzalez)
[16:02:38] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 10Observability-Metrics, 10SRE Observability (FY2024/2025-Q3): Remove librenms -> graphite integration, replace with gnmi - https://phabricator.wikimedia.org/T372457#10657690 (10cmooney) FWIW the gnmic export has been stable now for a while and we're beginning to use it...
[16:39:12] <wikibugs>	 (03CR) 10CI reject: [V:04-1] wmcs.openstack.tofu: when dealing with MR checkouts, operate on a different dir [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022) (owner: 10Arturo Borrero Gonzalez)
[16:40:11] <wikibugs>	 10wikitech.wikimedia.org, 10CAS-SSO, 10Phabricator, 07LDAP: Password reset not working for uid=maskaret,ou=people,dc=wikimedia,dc=org account - https://phabricator.wikimedia.org/T389496#10657815 (10bd808) Ok there is funkiness but it may just be confusion about SUL account vs Developer account naming. @Gry...
[16:40:52] <wikibugs>	 10wikitech.wikimedia.org, 10CAS-SSO, 10Phabricator, 07LDAP: Password reset not working as expected for Gryllida's Developer account - https://phabricator.wikimedia.org/T389496#10657818 (10bd808)
[16:41:53] <wikibugs>	 (03PS17) 10Arturo Borrero Gonzalez: wmcs.openstack.tofu: when dealing with MR checkouts, operate on a different dir [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022)
[19:59:47] <wikibugs>	 06Toolforge-standards-committee: Adoption request for "request" tool - https://phabricator.wikimedia.org/T389540 (10Tkarcher) 03NEW
[20:18:07] <wikibugs>	 (03open) 10dancy: IconLookup.vue: Fix .mw-prototyper-icon-lookup min-width [toolforge-repos/sitesampler] - 10https://gitlab.wikimedia.org/toolforge-repos/sitesampler/-/merge_requests/13
[20:18:09] <wikibugs>	 (03update) 10dancy: IconLookup.vue: Fix .mw-prototyper-icon-lookup min-width [toolforge-repos/sitesampler] - 10https://gitlab.wikimedia.org/toolforge-repos/sitesampler/-/merge_requests/13
[20:18:38] <wikibugs>	 (03update) 10dancy: IconLookup.vue: Fix .mw-prototyper-icon-lookup min-width [toolforge-repos/sitesampler] - 10https://gitlab.wikimedia.org/toolforge-repos/sitesampler/-/merge_requests/13
[20:26:05] <wikibugs>	 (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/11 (owner: 10l10n-bot)
[20:26:08] <wikibugs>	 (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/11 (owner: 10l10n-bot)
[20:32:22] <wikibugs>	 (03merge) 10annet: IconLookup.vue: Fix .mw-prototyper-icon-lookup min-width [toolforge-repos/sitesampler] - 10https://gitlab.wikimedia.org/toolforge-repos/sitesampler/-/merge_requests/13 (owner: 10dancy)
[20:38:35] <wikibugs>	 06Toolforge-standards-committee: Adoption request for "request" tool - https://phabricator.wikimedia.org/T389540#10659245 (10bd808) Problem 0 here is that I can't find anything that looks remotely like a license inside of /data/project/request and the tool pre-dates the selection of a default license at tool cre...
[20:40:42] <wikibugs>	 (03PS1) 10Andrew Bogott: Revert "Remove some networking panels" [openstack/horizon/horizon] (2024.1) - 10https://gerrit.wikimedia.org/r/1129932 (https://phabricator.wikimedia.org/T380531)
[20:41:11] <wikibugs>	 (03CR) 10Andrew Bogott: [V:03+2 C:03+2] Revert "Remove some networking panels" [openstack/horizon/horizon] (2024.1) - 10https://gerrit.wikimedia.org/r/1129932 (https://phabricator.wikimedia.org/T380531) (owner: 10Andrew Bogott)
[20:50:10] <wikibugs>	 06Toolforge-standards-committee: Adoption request for "request" tool - https://phabricator.wikimedia.org/T389540#10659302 (10Wurgl) bd808: ·The Problem with the tool show up when you do a  ` tail review-lag.err `  Table dewiki_p.flaggedpage_pending vanished. Whatever magic is needed, if this magic fixes the tool...
[21:21:50] <wikibugs>	 06Toolforge-standards-committee: Adoption request for "request" tool - https://phabricator.wikimedia.org/T389540#10659428 (10bd808) >>! In T389540#10659302, @Wurgl wrote: > bd808: ·The Problem with the tool show up when you do a  > ` > tail review-lag.err > ` >  > Table dewiki_p.flaggedpage_pending vanished. Wha...
[21:30:57] <wikibugs>	 (03PS1) 10Andrew Bogott: network_topology: disable VM creation from this panel [openstack/horizon/horizon] (2024.1) - 10https://gerrit.wikimedia.org/r/1129942
[21:31:44] <wikibugs>	 (03CR) 10Andrew Bogott: [V:03+2 C:03+2] network_topology: disable VM creation from this panel [openstack/horizon/horizon] (2024.1) - 10https://gerrit.wikimedia.org/r/1129942 (owner: 10Andrew Bogott)
[21:32:42] <wikibugs>	 06Toolforge-standards-committee: Adoption request for "request" tool - https://phabricator.wikimedia.org/T389540#10659524 (10bd808) >>! In T277883#9817112, @gerritbot wrote: > Change #1025821 **merged** by jenkins-bot: > %%%[mediawiki/extensions/FlaggedRevs@master] Get rid of flaggedpage_pending table%%% > https...
[21:43:55] <wikibugs>	 06cloud-services-team, 10Horizon, 05Cloud-Services-Origin-User, 13Patch-For-Review: /project/instances/{instance id}/?tab=instance_details__interfaces returns HTTP 500; Interfaces tab never finishes loading - https://phabricator.wikimedia.org/T380531#10659595 (10Andrew) That panel has a weird cross-panel r...
[21:52:43] <wikibugs>	 10wikitech.wikimedia.org, 06MediaWiki-Platform-Team, 05SUL3: Fix stuck old cookies on Wikitech - https://phabricator.wikimedia.org/T389433#10659681 (10Tgr) 05Open→03Resolved a:03Tgr Stuck cookies are cleared now. I didn't have a real session to test with, so I recreated the stuck cookies manually,...
[22:06:10] <wikibugs>	 06Toolforge-standards-committee: Adoption request for "request" tool - https://phabricator.wikimedia.org/T389540#10659779 (10bd808) Warning code of unknown license ahead. `lang=python,name=     #########################     # select and insert lag #     #########################      bot_wiki.query('select count...
[22:06:56] <jinxer-wm>	 FIRING: SystemdUnitDown: The service unit drain_rabbitmq_notification_error.service is in failed status on host cloudrabbit1001. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudrabbit1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[22:11:56] <jinxer-wm>	 RESOLVED: SystemdUnitDown: The service unit drain_rabbitmq_notification_error.service is in failed status on host cloudrabbit1001. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudrabbit1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[23:12:48] <wikibugs>	 10wikitech.wikimedia.org, 10CAS-SSO, 10Phabricator, 07LDAP: Password reset not working as expected for Gryllida's Developer account - https://phabricator.wikimedia.org/T389496#10659986 (10bd808) 05Open→03Invalid `lang=irc [18:33]  <      gry> bd808: thanks, i was able to login as you told me the  w...