[09:19:33] 10serviceops, 10Infrastructure-Foundations, 10netbox: Netbox and Redis - https://phabricator.wikimedia.org/T311385 (10akosiaris) >>! In T311385#8134119, @ayounsi wrote: >>> Thanks for this task. So, from what I gathered, netbox uses Redis for caching and task queuing purposes, and support using different dat... [09:25:51] 10serviceops, 10Prod-Kubernetes: Clean up profile::docker::storage from puppet - https://phabricator.wikimedia.org/T315977 (10Clement_Goubert) a:03Clement_Goubert [09:34:01] 10serviceops, 10Prod-Kubernetes: Clean up profile::docker::storage from puppet - https://phabricator.wikimedia.org/T315977 (10Clement_Goubert) [x] Checking hiera does not define `profile::docker::storage::(vg_to_remove|physical_volumes)`: ` ❯ rgrep -E 'profile::docker::storage::(vg|physical)' hieradata hierad... [09:36:26] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Update Kubernetes clusters to v1.23 - https://phabricator.wikimedia.org/T307943 (10JMeybohm) [09:52:42] 10serviceops, 10Prod-Kubernetes: Clean up profile::docker::storage from puppet - https://phabricator.wikimedia.org/T315977 (10Clement_Goubert) p:05Triage→03Low [10:08:45] 10serviceops, 10Diffusion, 10Gerrit, 10serviceops-collab, 10Patch-For-Review: Gerrit replication to codfw (gerrit-replica.wikimedia.org) stopped working after Gerrit 3.4.5 upgrade - https://phabricator.wikimedia.org/T315942 (10hashar) I kept digging into it last night and this morning. Based on https://... [10:16:04] 10serviceops, 10Diffusion, 10Gerrit, 10serviceops-collab, 10Patch-For-Review: Gerrit replication to codfw (gerrit-replica.wikimedia.org) stopped working after Gerrit 3.4.5 upgrade - https://phabricator.wikimedia.org/T315942 (10hashar) 05Resolved→03Open [12:02:10] 10serviceops, 10Prod-Kubernetes, 10Patch-For-Review: Clean up profile::docker::storage from puppet - https://phabricator.wikimedia.org/T315977 (10Clement_Goubert) Ok, all changes set up for cleanup. LMK. [12:53:43] 10serviceops, 10Diffusion, 10Gerrit, 10serviceops-collab, and 2 others: Gerrit replication to codfw (gerrit-replica.wikimedia.org) stopped working after Gerrit 3.4.5 upgrade - https://phabricator.wikimedia.org/T315942 (10hashar) [12:53:53] 10serviceops, 10Diffusion, 10Gerrit, 10serviceops-collab, and 2 others: Gerrit replication to codfw (gerrit-replica.wikimedia.org) stopped working after Gerrit 3.4.5 upgrade - https://phabricator.wikimedia.org/T315942 (10hashar) Filed as https://bugs.chromium.org/p/gerrit/issues/detail?id=16215 . An ideal... [13:41:56] 10serviceops, 10Infrastructure-Foundations, 10netbox: Netbox and Redis - https://phabricator.wikimedia.org/T311385 (10ayounsi) > Can netbox be configured without the redis cache? After all it just might not be worth it to have it. Unfortunately not. According to https://docs.netbox.dev/en/stable/configuratio... [20:04:21] 10serviceops, 10Data-Persistence, 10Phabricator, 10serviceops-collab, 10Release-Engineering-Team (Bonus Level 🕹ī¸): sort out mysql privileges for phab1004/phab2002 - https://phabricator.wikimedia.org/T315713 (10Dzahn) [20:15:19] 10serviceops, 10Data-Persistence, 10Phabricator, 10serviceops-collab, 10Release-Engineering-Team (Bonus Level 🕹ī¸): sort out mysql privileges for phab1004/phab2002 - https://phabricator.wikimedia.org/T315713 (10Dzahn) Hello Data-Persistence, we have the existing Phabricator servers `phab1001.eqiad.wmnet`... [20:17:43] 10serviceops, 10Phabricator, 10serviceops-collab, 10Patch-For-Review, 10Release-Engineering-Team (Bonus Level 🕹ī¸): move phabricator to new hardware generation - https://phabricator.wikimedia.org/T280597 (10Dzahn) @thcipriani contacting data persistence about DB privileges at T315713#8183037. [20:25:19] 10serviceops, 10Data-Persistence, 10Phabricator, 10serviceops-collab, 10Release-Engineering-Team (Bonus Level 🕹ī¸): sort out mysql privileges for phab1004/phab2002 - https://phabricator.wikimedia.org/T315713 (10Dzahn) 05Open→03In progress [20:25:29] 10serviceops, 10Phabricator, 10serviceops-collab, 10Patch-For-Review, 10Release-Engineering-Team (Bonus Level 🕹ī¸): move phabricator to new hardware generation - https://phabricator.wikimedia.org/T280597 (10Dzahn) [20:41:57] 10serviceops, 10SRE, 10Sustainability (Incident Followup): Set API and appserver weights in eqiad - https://phabricator.wikimedia.org/T304800 (10Dzahn) [20:42:53] 20:42 <+icinga-wm> RECOVERY - Check systemd state on otrs1001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [20:42:57] fixed that [21:27:33] 10serviceops, 10SRE, 10Sustainability (Incident Followup): Set API and appserver weights in eqiad - https://phabricator.wikimedia.org/T304800 (10Dzahn) @RLazarus Thank you! Done! I changed the value from 30 to 25 for any server within the range mw1307 through mw1348 that was either appserver or api_appserve... [21:29:43] 10serviceops, 10SRE, 10Sustainability (Incident Followup): Set API and appserver weights in eqiad - https://phabricator.wikimedia.org/T304800 (10Dzahn) 05Open→03Resolved a:03Dzahn [21:57:00] 10serviceops, 10Phabricator, 10serviceops-collab, 10Release-Engineering-Team (Bonus Level 🕹ī¸): Email tool maintainers about git-ssh deprecation on phabricator - https://phabricator.wikimedia.org/T313359 (10thcipriani) Email draft: ` --- Subject: Phabricator's git over ssh will fail after $DATE --- As of... [23:20:58] 10serviceops, 10Phabricator, 10serviceops-collab, 10Release-Engineering-Team (Bonus Level 🕹ī¸): Email tool maintainers about git-ssh deprecation on phabricator - https://phabricator.wikimedia.org/T313359 (10Dzahn) @thcipriani Sounds good to me. It links to the docs how to push over https, how to move to g...