Fork me on GitHub

Wikimedia IRC logs browser - #wikimedia-releng

Filter:
Start date
End date

Displaying 97 items:

2026-03-15 01:46:32 <wmcs-alerts> FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
2026-03-15 01:56:32 <wmcs-alerts> RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
2026-03-15 02:41:32 <wmcs-alerts> FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
2026-03-15 02:51:32 <wmcs-alerts> RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
2026-03-15 02:57:56 <eileen> I think we have someone maliciously meddling with phabs - last time I saw this I feel like it was a clue to some bigger problem https://phabricator.wikimedia.org/p/BucheonFC/
2026-03-15 03:25:59 <p858snake|cloud> they are already disabled on phab and locked in SUL
2026-03-15 03:26:06 <p858snake|cloud> just pending a phab admin to cleanup the comments
2026-03-15 03:31:14 <eileen> cool
2026-03-15 03:45:32 <wmcs-alerts> FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
2026-03-15 03:47:36 <wikibugs> 'Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132 (''Novem_Linguae) ''NEW'
2026-03-15 03:48:55 <wikibugs> 'Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11710968 (''Peachey88)'
2026-03-15 03:48:58 <wikibugs> 'Phabricator: Make sure anti-vandalism features are up to snuff - https://phabricator.wikimedia.org/T84#11710969 (''Peachey88)'
2026-03-15 03:51:40 <wikibugs> 'Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11710970 (''Novem_Linguae)'
2026-03-15 03:57:04 <wikibugs> 'Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11710973 (''Pppery) This is why we can't have nice things.'
2026-03-15 04:00:32 <wmcs-alerts> RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
2026-03-15 04:00:34 <wikibugs> 'Project-Admins: Create project tag for <INSERT HERE> - https://phabricator.wikimedia.org/T420135 (''260315t1244) ''NEW'
2026-03-15 04:01:23 <wikibugs> 'Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711008 (''260315t1244) https://www.youtube.com/watch?v=oUZttxRcPZw'
2026-03-15 04:26:53 <wikibugs> 'Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711020 (''SldrF)'
2026-03-15 04:27:06 <wikibugs> 'Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711021 (''SldrF) ''Open''Declined'
2026-03-15 04:27:38 <wikibugs> 'Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711023 (''Pppery) ''Declined''Open'
2026-03-15 04:27:42 <wikibugs> 'Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711025 (''Pppery)'
2026-03-15 04:29:34 <wikibugs> 'Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711026 (''Novem_Linguae)'
2026-03-15 04:30:21 <wikibugs> 'Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711028 (''Izno) > it is super easy to get into the Trusted-Contributors group. It is. And it isn't. You have to know someone who is in it to be added to it because ther...'
2026-03-15 04:35:07 <wikibugs> 'Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711030 (''Pppery) Or file a Phab task requesting you be added. That's been done at least once (T361392) (And I also filed a task requesting myself be added to the rela...'
2026-03-15 04:44:32 <wmcs-alerts> FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
2026-03-15 04:54:32 <wmcs-alerts> RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
2026-03-15 05:46:41 <wikibugs> 'Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711037 (''Bugreporter) > assignee Note new Outreachy/GSoC users may want to claim some tasks themselves. It can instead be rate limited.'
2026-03-15 05:47:32 <wmcs-alerts> FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
2026-03-15 05:48:33 <wikibugs> 'Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711038 (''Pppery) ideally the Outreachy/GSOC mentor would add their mentees to #trusted-contributors'
2026-03-15 05:54:25 <wikibugs> 'Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711039 (''Bugreporter) > I'm also not convinced phab-ban access should be extended more widely. Right now the only safeguard it has is that you can't disable admins, an...'
2026-03-15 06:07:35 <wikibugs> 'Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711042 (''Peachey88) Phab-Ban is not within the scope of this task, Lets take it elsewhere, such as dedicated task if required?'
2026-03-15 06:12:32 <wmcs-alerts> RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
2026-03-15 06:15:32 <wmcs-alerts> FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
2026-03-15 06:20:26 <wikibugs> 'Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711047 (''Novem_Linguae) (Phab-Ban worked fantastic tonight. Volunteers were able to handle the Phabricator vandalism incident at a time when most WMFers were probably...'
2026-03-15 06:25:32 <wmcs-alerts> RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
2026-03-15 06:32:39 <wikibugs> 'Release-Engineering-Team, ''ChangeProp, ''Data-Engineering, ''EventStreams, and 15 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#11711052 (''Kgarcia181) ''Open''In progress p:''Medium''Unbreak! a:''Kgarcia181'
2026-03-15 06:35:54 <wikibugs> 'Release-Engineering-Team, ''ChangeProp, ''Data-Engineering, ''EventStreams, and 15 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#11711055 (''Peachey88) ''In progress''Open p:''Unbreak!''Medium a:''Kgarcia181''None'
2026-03-15 07:09:27 <wikibugs> 'Phabricator, ''Tool-phab-ban: Temporary ban feature from phab-ban to quickly response to Phabricator vandalism - https://phabricator.wikimedia.org/T420136 (''Bugreporter) ''NEW'
2026-03-15 07:17:53 <wikibugs> 'Phabricator, ''Tool-phab-ban: Temporary ban feature from phab-ban to quickly response to Phabricator vandalism - https://phabricator.wikimedia.org/T420136#11711080 (''Novem_Linguae) Is the status quo not working well? I feel the 85 existing phab-banners handled the vandalism incident tonight well. https://w...'
2026-03-15 07:21:48 <wikibugs> 'Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711083 (''Escargot_rouge) Preventing the editing of subscribers would likely prevent mentioning, since mentioning someone automatically adds them as a subscriber.'
2026-03-15 07:26:25 <wikibugs> 'Phabricator, ''Tool-phab-ban: Temporary ban feature from phab-ban to quickly response to Phabricator vandalism - https://phabricator.wikimedia.org/T420136#11711086 (''Peachey88) Can you provide a rationale, if someone is blocked for vandalism that it would ideal for it to be only temporary?'
2026-03-15 07:34:26 <wikibugs> 'Phabricator, ''Tool-phab-ban: Temporary ban feature from phab-ban to quickly response to Phabricator vandalism - https://phabricator.wikimedia.org/T420136#11711093 (''Bugreporter) ''Open''Declined I withdraw this task: previously I think other user's concern is the phab-ban tool (or current acl*user...'
2026-03-15 07:41:32 <wmcs-alerts> FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
2026-03-15 07:46:32 <wmcs-alerts> RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
2026-03-15 09:07:17 <wikibugs> 'Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711117 (''ScottishFimishRadish) ''Open''Declined'
2026-03-15 09:07:24 <wikibugs> 'Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711120 (''ScottishFimishRadish) ㅋㅋㅋㅋ'
2026-03-15 09:08:51 <wikibugs> 'Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711121 (''A_smart_kitten) ''Declined''Open'
2026-03-15 09:15:42 <wikibugs> 'Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711124 (''Primefac1)'
2026-03-15 09:15:55 <wikibugs> 'Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711125 (''Primefac1) ''Open''Resolved p:''Triage''Low a:''Primefac1'
2026-03-15 09:17:06 <wikibugs> 'Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711131 (''Novem_Linguae) ''Resolved''Open p:''Low''Triage a:''Primefac1''None'
2026-03-15 09:17:38 <wikibugs> 'Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711137 (''Novem_Linguae)'
2026-03-15 09:23:55 <wikibugs> 'Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711140 (''BucheonCityHall6)'
2026-03-15 09:23:57 <wikibugs> 'Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711143 (''BucheonCityHall6) ''Open''Declined'
2026-03-15 09:28:07 <wikibugs> 'Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711152 (''Novem_Linguae) ''Declined''Open'
2026-03-15 09:28:25 <wikibugs> 'Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711155 (''Novem_Linguae)'
2026-03-15 09:41:32 <wmcs-alerts> FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
2026-03-15 09:46:32 <wmcs-alerts> RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
2026-03-15 10:40:32 <wmcs-alerts> FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
2026-03-15 10:41:54 <wikibugs> ('Abandoned) ''Hashar: build: Updating flatted to 3.4.1 [performance/WikimediaDebug] - ''https://gerrit.wikimedia.org/r/1252408 (owner: ''Libraryupgrader)'
2026-03-15 10:43:06 <wikibugs> ('Abandoned) ''Hashar: build: Updating flatted to 3.4.1 [integration/docroot] - ''https://gerrit.wikimedia.org/r/1251876 (owner: ''Libraryupgrader)'
2026-03-15 10:45:32 <wmcs-alerts> RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
2026-03-15 11:46:59 <wikibugs> 'Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711356 (''Bucheon606) DON'T BE A RACIST.'
2026-03-15 11:52:25 <hashar> Mar 15 11:39:38 deployment-sessionstore06 systemd[1]: cassandra.service: Failed with result 'oom-kill'.
2026-03-15 11:52:25 <hashar> the instance has 2G, Cassandra takes 1.3G
2026-03-15 11:52:35 <hashar> and it seems to line up with puppet agent running every 30 minutes
2026-03-15 11:53:51 <wikibugs> 'Beta-Cluster-Infrastructure, ''Cassandra, ''Data-Persistence: Cassandra killed by oom-killer and prometheus scrapes failing intermittently on deployment-sessionstore06 - https://phabricator.wikimedia.org/T415021#11711360 (''hashar)'
2026-03-15 11:53:55 <wikibugs> 'Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-sessionstore06 is down - https://phabricator.wikimedia.org/T420071#11711362 (''hashar) →''Duplicate dup:''T415021'
2026-03-15 11:53:56 <hashar> which is https://phabricator.wikimedia.org/T415021 :)
2026-03-15 12:07:56 <wikibugs> 'Beta-Cluster-Infrastructure, ''Cassandra, ''Data-Persistence: Cassandra killed by oom-killer and prometheus scrapes failing intermittently on deployment-sessionstore06 - https://phabricator.wikimedia.org/T415021#11711367 (''hashar) It lines up with `puppet agent` running every 30 minutes on the host possib...'
2026-03-15 13:07:32 <wmcs-alerts> FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
2026-03-15 13:07:37 <wikibugs> 'Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-sessionstore06 is down - https://phabricator.wikimedia.org/T420139 (''wmcs-alerts) ''NEW'
2026-03-15 13:12:32 <wmcs-alerts> RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
2026-03-15 13:47:07 <wikibugs> 'Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711480 (''Tgr) I wonder if we could auto-add people to Trusted-Contributors based on some modest contribution criteria (like >1000 global wiki edits)?'
2026-03-15 13:54:40 <wikibugs> 'Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11711512 (''A_smart_kitten) On the subject of what's been proposed -- I suppose I worry in case this sort of action, if taken, might be a 'reflex-reaction' to a recent bo...'
2026-03-15 14:02:32 <wmcs-alerts> FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
2026-03-15 14:07:32 <wmcs-alerts> RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
2026-03-15 14:17:32 <wmcs-alerts> FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
2026-03-15 14:22:32 <wmcs-alerts> RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
2026-03-15 14:31:32 <wmcs-alerts> FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-kafka-logging01 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
2026-03-15 14:40:32 <wmcs-alerts> FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
2026-03-15 14:45:32 <wmcs-alerts> RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
2026-03-15 15:11:32 <wmcs-alerts> FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
2026-03-15 15:16:37 <wmcs-alerts> RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
2026-03-15 16:40:32 <wmcs-alerts> FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
2026-03-15 16:45:32 <wmcs-alerts> RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
2026-03-15 16:56:18 <wikibugs> 'Phabricator: Be able to turn off account approval email - https://phabricator.wikimedia.org/T420144 (''Reedy) ''NEW'
2026-03-15 17:34:42 <A_smart_kitten> wonders if there's a way to 'downtime' specific alerts from wmcs-alerts in the same/similar way to (they assume) alerts in production are able to be downtimed
2026-03-15 17:35:17 <taavi> https://wikitech.wikimedia.org/wiki/Help:Cloud_VPS_managed_monitoring#Alerts
2026-03-15 18:02:20 <A_smart_kitten> thanks taavi! that answers my wondering... "Members and readers of a project can set silences for alerts in projects they have access to."
2026-03-15 19:40:32 <wmcs-alerts> FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
2026-03-15 19:50:32 <wmcs-alerts> RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
2026-03-15 20:58:10 <wikibugs> 'Phabricator, ''Release-Engineering-Team (Priority Backlog 📥): Update to Phorge/Arcanist upstream (2026.xx) - https://phabricator.wikimedia.org/T410849#11711872 (''Aklapper) Temporary comment: Should pull Phorge's 9ae0020a857e, the commit after seems to be buggy'
2026-03-15 21:07:39 <wikibugs> 'Phabricator (Upstream), ''Upstream: Be able to turn off account approval email - https://phabricator.wikimedia.org/T420144#11711886 (''Aklapper) Currently you can either filter them in your email client, or ask to be removed as a Phab admin.'
2026-03-15 21:22:01 <wikibugs> 'Phabricator (Upstream), ''Upstream: Be able to turn off account approval email - https://phabricator.wikimedia.org/T420144#11711912 (''Aklapper)'
2026-03-15 22:10:32 <wmcs-alerts> FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
2026-03-15 22:30:32 <wmcs-alerts> RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
2026-03-15 23:42:59 <zabe> Someone else also no longer gets the "classical home screen" at /app/home when going to logstash? I do now get redirected to /app/opensearch_dashboards_overview. Is that something I can change back for my account?

This page is generated from SQL logs, you can also download static txt files from here