Depends on your servers available memory limit.
Typically; in my opinion anything over 256MB is bad news. Plugins make Wordpress memory hungry and poorly designed plugins cause problems.
In the event that website suffers a brute force ofnsay xmlrpc or wplogin; a memory limit of 512MB is going to hurt you as it can quickly cause your server to experience out of memory errors.
You’re basically giving permission for it to use up too that amount; so each incoming request that requires an execution of php can hit that amount.
One of the downsides of having high memory limits.