[01:46:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [01:56:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [02:41:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [02:51:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [02:57:56] I think we have someone maliciously meddling with phabs - last time I saw this I feel like it was a clue to some bigger problem https://phabricator.wikimedia.org/p/BucheonFC/ [03:25:59] they are already disabled on phab and locked in SUL [03:26:06] just pending a phab admin to cleanup the comments [03:31:14] cool [03:45:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [03:47:36] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132 (10Novem_Linguae) 03NEW [03:48:55] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11710968 (10Peachey88) [03:48:58] 10Phabricator: Make sure anti-vandalism features are up to snuff - https://phabricator.wikimedia.org/T84#11710969 (10Peachey88) [03:51:40] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11710970 (10Novem_Linguae) [03:57:04] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11710973 (10Pppery) This is why we can't have nice things. [04:00:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [04:00:34] 06Project-Admins: Create project tag for - https://phabricator.wikimedia.org/T420135 (10260315t1244) 03NEW [04:01:23] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711008 (10260315t1244) https://www.youtube.com/watch?v=oUZttxRcPZw [04:26:53] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711020 (10SldrF) [04:27:06] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711021 (10SldrF) 05Open→03Declined [04:27:38] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711023 (10Pppery) 05Declined→03Open [04:27:42] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711025 (10Pppery) [04:29:34] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711026 (10Novem_Linguae) [04:30:21] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711028 (10Izno) > it is super easy to get into the Trusted-Contributors group. It is. And it isn't. You have to know someone who is in it to be added to it because ther... [04:35:07] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711030 (10Pppery) Or file a Phab task requesting you be added. That's been done at least once (T361392) (And I also filed a task requesting myself be added to the rela... [04:44:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [04:54:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [05:46:41] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711037 (10Bugreporter) > assignee Note new Outreachy/GSoC users may want to claim some tasks themselves. It can instead be rate limited. [05:47:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [05:48:33] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711038 (10Pppery) ideally the Outreachy/GSOC mentor would add their mentees to #trusted-contributors [05:54:25] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711039 (10Bugreporter) > I'm also not convinced phab-ban access should be extended more widely. Right now the only safeguard it has is that you can't disable admins, an... [06:07:35] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711042 (10Peachey88) Phab-Ban is not within the scope of this task, Lets take it elsewhere, such as dedicated task if required? [06:12:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [06:15:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [06:20:26] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711047 (10Novem_Linguae) (Phab-Ban worked fantastic tonight. Volunteers were able to handle the Phabricator vandalism incident at a time when most WMFers were probably... [06:25:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [06:32:39] 06Release-Engineering-Team, 10ChangeProp, 06Data-Engineering, 10EventStreams, and 15 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#11711052 (10Kgarcia181) 05Open→03In progress p:05Medium→03Unbreak! a:03Kgarcia181 [06:35:54] 06Release-Engineering-Team, 10ChangeProp, 06Data-Engineering, 10EventStreams, and 15 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#11711055 (10Peachey88) 05In progress→03Open p:05Unbreak!→03Medium a:05Kgarcia181→03None [07:09:27] 10Phabricator, 10Tool-phab-ban: Temporary ban feature from phab-ban to quickly response to Phabricator vandalism - https://phabricator.wikimedia.org/T420136 (10Bugreporter) 03NEW [07:17:53] 10Phabricator, 10Tool-phab-ban: Temporary ban feature from phab-ban to quickly response to Phabricator vandalism - https://phabricator.wikimedia.org/T420136#11711080 (10Novem_Linguae) Is the status quo not working well? I feel the 85 existing phab-banners handled the vandalism incident tonight well. https://w... [07:21:48] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711083 (10Escargot_rouge) Preventing the editing of subscribers would likely prevent mentioning, since mentioning someone automatically adds them as a subscriber. [07:26:25] 10Phabricator, 10Tool-phab-ban: Temporary ban feature from phab-ban to quickly response to Phabricator vandalism - https://phabricator.wikimedia.org/T420136#11711086 (10Peachey88) Can you provide a rationale, if someone is blocked for vandalism that it would ideal for it to be only temporary? [07:34:26] 10Phabricator, 10Tool-phab-ban: Temporary ban feature from phab-ban to quickly response to Phabricator vandalism - https://phabricator.wikimedia.org/T420136#11711093 (10Bugreporter) 05Open→03Declined I withdraw this task: previously I think other user's concern is the phab-ban tool (or current acl*user... [07:41:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [07:46:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [09:07:17] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711117 (10ScottishFimishRadish) 05Open→03Declined [09:07:24] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711120 (10ScottishFimishRadish) ㅋㅋㅋㅋ [09:08:51] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711121 (10A_smart_kitten) 05Declined→03Open [09:15:42] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711124 (10Primefac1) [09:15:55] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711125 (10Primefac1) 05Open→03Resolved p:05Triage→03Low a:03Primefac1 [09:17:06] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711131 (10Novem_Linguae) 05Resolved→03Open p:05Low→03Triage a:05Primefac1→03None [09:17:38] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711137 (10Novem_Linguae) [09:23:55] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711140 (10BucheonCityHall6) [09:23:57] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711143 (10BucheonCityHall6) 05Open→03Declined [09:28:07] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711152 (10Novem_Linguae) 05Declined→03Open [09:28:25] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711155 (10Novem_Linguae) [09:41:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [09:46:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [10:40:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [10:41:54] (03Abandoned) 10Hashar: build: Updating flatted to 3.4.1 [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1252408 (owner: 10Libraryupgrader) [10:43:06] (03Abandoned) 10Hashar: build: Updating flatted to 3.4.1 [integration/docroot] - 10https://gerrit.wikimedia.org/r/1251876 (owner: 10Libraryupgrader) [10:45:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [11:46:59] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711356 (10Bucheon606) DON'T BE A RACIST. [11:52:25] Mar 15 11:39:38 deployment-sessionstore06 systemd[1]: cassandra.service: Failed with result 'oom-kill'. [11:52:25] the instance has 2G, Cassandra takes 1.3G [11:52:35] and it seems to line up with puppet agent running every 30 minutes [11:53:51] 10Beta-Cluster-Infrastructure, 10Cassandra, 06Data-Persistence: Cassandra killed by oom-killer and prometheus scrapes failing intermittently on deployment-sessionstore06 - https://phabricator.wikimedia.org/T415021#11711360 (10hashar) [11:53:55] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-sessionstore06 is down - https://phabricator.wikimedia.org/T420071#11711362 (10hashar) →14Duplicate dup:03T415021 [11:53:56] which is https://phabricator.wikimedia.org/T415021 :) [12:07:56] 10Beta-Cluster-Infrastructure, 10Cassandra, 06Data-Persistence: Cassandra killed by oom-killer and prometheus scrapes failing intermittently on deployment-sessionstore06 - https://phabricator.wikimedia.org/T415021#11711367 (10hashar) It lines up with `puppet agent` running every 30 minutes on the host possib... [13:07:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [13:07:37] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-sessionstore06 is down - https://phabricator.wikimedia.org/T420139 (10wmcs-alerts) 03NEW [13:12:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [13:47:07] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711480 (10Tgr) I wonder if we could auto-add people to Trusted-Contributors based on some modest contribution criteria (like >1000 global wiki edits)? [13:54:40] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711512 (10A_smart_kitten) On the subject of what's been proposed -- I suppose I worry in case this sort of action, if taken, might be a 'reflex-reaction' to a recent bo... [14:02:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:07:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:17:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:22:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:31:32] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-kafka-logging01 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [14:40:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:45:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [15:11:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [15:16:37] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [16:40:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [16:45:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [16:56:18] 10Phabricator: Be able to turn off account approval email - https://phabricator.wikimedia.org/T420144 (10Reedy) 03NEW [17:34:42] * A_smart_kitten wonders if there's a way to 'downtime' specific alerts from wmcs-alerts in the same/similar way to (they assume) alerts in production are able to be downtimed [17:35:17] https://wikitech.wikimedia.org/wiki/Help:Cloud_VPS_managed_monitoring#Alerts [18:02:20] thanks taavi! that answers my wondering... "Members and readers of a project can set silences for alerts in projects they have access to." [19:40:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [19:50:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [20:58:10] 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥): Update to Phorge/Arcanist upstream (2026.xx) - https://phabricator.wikimedia.org/T410849#11711872 (10Aklapper) Temporary comment: Should pull Phorge's 9ae0020a857e, the commit after seems to be buggy [21:07:39] 10Phabricator (Upstream), 07Upstream: Be able to turn off account approval email - https://phabricator.wikimedia.org/T420144#11711886 (10Aklapper) Currently you can either filter them in your email client, or ask to be removed as a Phab admin. [21:22:01] 10Phabricator (Upstream), 07Upstream: Be able to turn off account approval email - https://phabricator.wikimedia.org/T420144#11711912 (10Aklapper) [22:10:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [22:30:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [23:42:59] Someone else also no longer gets the "classical home screen" at /app/home when going to logstash? I do now get redirected to /app/opensearch_dashboards_overview. Is that something I can change back for my account?