[05:30:28] 10serviceops, 10Phabricator, 10serviceops-collab, 10Patch-For-Review, 10Release-Engineering-Team (Bonus Level 🕹ī¸): sort out mysql privileges for phab1004/phab2002 - https://phabricator.wikimedia.org/T315713 (10Marostegui) Excellent - thank you Daniel [07:34:39] 10serviceops, 10Infrastructure-Foundations, 10netbox: Netbox and Redis - https://phabricator.wikimedia.org/T311385 (10ayounsi) Awesome, thanks! Then let's stick to the plan of of remote Redis, as a risk of higher latency is better than a risk of split view :) And it's easier to configure. [08:58:45] 10serviceops, 10GitLab (Infrastructure): Reduce usage of public IPv4 addresses on GitLab hosts - https://phabricator.wikimedia.org/T310265 (10ayounsi) [10:52:47] 10serviceops, 10Patch-For-Review: Put parse parse10[01-24] in production - https://phabricator.wikimedia.org/T307219 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=6ae25b9f-a382-4184-824f-2ae58210788f) set by cgoubert@cumin1001 for 7 days, 0:00:00 on 3 host(s) and their services with reaso... [11:00:37] 10serviceops, 10Patch-For-Review: Put parse parse10[01-24] in production - https://phabricator.wikimedia.org/T307219 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=ceb79fae-b2a5-4618-9cd0-d21af9cca032) set by cgoubert@cumin1001 for 7 days, 0:00:00 on 7 host(s) and their services with reaso... [11:56:37] 10serviceops, 10API Platform, 10Growth-Structured-Tasks, 10Image-Suggestions, and 5 others: GrowthExperiments\NewcomerTasks\AddImage\ServiceImageRecommendationProvider::get Unable to decode JSON response for page {title} upstream connect error or disconnect/reset b... - https://phabricator.wikimedia.org/T313973 [12:04:29] 10serviceops, 10API Platform, 10Growth-Structured-Tasks, 10Image-Suggestions, and 5 others: GrowthExperiments\NewcomerTasks\AddImage\ServiceImageRecommendationProvider::get Unable to decode JSON response for page {title} upstream connect error or disconnect/reset b... - https://phabricator.wikimedia.org/T313973 [12:33:27] 10serviceops, 10Observability-Alerting, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Migrate kubernetes alerts away from icinga - https://phabricator.wikimedia.org/T311251 (10JMeybohm) [13:08:42] 10serviceops, 10Machine-Learning-Team, 10ORES: Reimage rdb1009, rdb1010 as bullseye - https://phabricator.wikimedia.org/T317189 (10akosiaris) [13:09:01] 10serviceops, 10Machine-Learning-Team, 10ORES: Reimage rdb1009, rdb1010 as bullseye - https://phabricator.wikimedia.org/T317189 (10akosiaris) p:05Triage→03High [13:15:12] akosiaris: o/ does https://phabricator.wikimedia.org/T317189 need some input from the ml team? (I saw the ORES tag) [13:18:51] elukey: no, I think not. I mostly added it for historical reasons (that might have been a mistake) [13:19:10] in fact, I think all that is needed is to set the new debian distro and reimage them [13:19:23] akosiaris: super thanks [13:36:41] 10serviceops, 10Machine-Learning-Team, 10ORES: Reimage rdb1009, rdb1010 as bullseye - https://phabricator.wikimedia.org/T317189 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1001 for host rdb1009.eqiad.wmnet with OS bullseye [14:08:23] 10serviceops, 10Machine-Learning-Team, 10ORES: Reimage rdb1009, rdb1010 as bullseye - https://phabricator.wikimedia.org/T317189 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1001 for host rdb1009.eqiad.wmnet with OS bullseye completed: - rdb1009 (**PASS**) - Downtime... [14:23:12] 10serviceops, 10Machine-Learning-Team, 10ORES: Reimage rdb1009, rdb1010 as bullseye - https://phabricator.wikimedia.org/T317189 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1001 for host rdb1010.eqiad.wmnet with OS bullseye [14:24:13] 10serviceops, 10Observability-Alerting, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Migrate kubernetes alerts away from icinga - https://phabricator.wikimedia.org/T311251 (10JMeybohm) [14:41:50] 10serviceops, 10Observability-Alerting, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Migrate kubernetes alerts away from icinga - https://phabricator.wikimedia.org/T311251 (10akosiaris) > Note: I don't understand why exec_sync is excluded here as that does not go above 240ms over the last 90days.... [14:54:50] 10serviceops, 10Machine-Learning-Team, 10ORES: Reimage rdb1009, rdb1010 as bullseye - https://phabricator.wikimedia.org/T317189 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1001 for host rdb1010.eqiad.wmnet with OS bullseye completed: - rdb1010 (**PASS**) - Downtime... [14:57:01] 10serviceops, 10Machine-Learning-Team, 10ORES: Reimage rdb1009, rdb1010 as bullseye - https://phabricator.wikimedia.org/T317189 (10akosiaris) 05Open→03Resolved Hosts reimaged. aside from a small backlog for changeprop and an increase in latency for api-gateway for a bit, no other side-effect. Closing thi... [15:07:08] 10serviceops, 10Machine-Learning-Team, 10ORES: Reimage rdb1009, rdb1010 as bullseye - https://phabricator.wikimedia.org/T317189 (10MoritzMuehlenhoff) Thanks :-) [15:10:26] 10serviceops: Put parse parse10[01-24] in production - https://phabricator.wikimedia.org/T307219 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=abae9736-4890-49f5-ae11-999ff8a0514e) set by cgoubert@cumin1001 for 7 days, 0:00:00 on 3 host(s) and their services with reason: Downtiming replaced... [15:17:37] 10serviceops: Put parse parse10[01-24] in production - https://phabricator.wikimedia.org/T307219 (10Clement_Goubert) `parse1013.eqiad.wmnet` replaced `wtp1046.eqiad.wmnet` `parse1014.eqiad.wmnet` replaced `wtp1047.eqiad.wmnet` `parse1015.eqiad.wmnet` replaced `wtp1048.eqiad.wmnet` `parse1016.eqiad.wmnet` replace... [17:03:38] 10serviceops, 10API Platform, 10Growth-Structured-Tasks, 10Image-Suggestions, and 5 others: GrowthExperiments\NewcomerTasks\AddImage\ServiceImageRecommendationProvider::get Unable to decode JSON response for page {title} upstream connect error or disconnect/reset b... - https://phabricator.wikimedia.org/T313973 [21:06:35] 10serviceops, 10API Platform, 10Growth-Structured-Tasks, 10Image-Suggestions, and 7 others: GrowthExperiments\NewcomerTasks\AddImage\ServiceImageRecommendationProvider::get Unable to decode JSON response for page {title} upstream connect error or disconnect/reset b... - https://phabricator.wikimedia.org/T313973