Hi Pavel,
A couple of things - Firstly, are you opening each instance of VI as a direct plug in to your tracks in Logic or using VEnsmble? Though this won't have a direct affect on your RAM in the same way, if I understand the workings correctly, each instance will require RAM to run, so the more tracks, the more RAM, but it might not show in the same way. It might be worth considering opening instances in the 64bit version of Ensemble as this will allow each VI instance to access all of your available RAM while using less for itself. (Ithink I'm right in this - if not please correct me. Also, apologies as this seems overly complicated.)
Secondly, are you definitely working in 64 bit mode in Logic? This is a common mistake which is easily solved - it took me a year to notice I was doing it wrong! Before opening Logic, find it in your applications folder and open the info window (Apple + I). In there is a little button that chooses whether you are opening in 32 bit or 64 bit mode. Make sure you're in 64 bit mode as this will allow your mac to access more than 4GB of RAM. The disadvantage of 64bit mode is that you cannot export your audio to movie - so you have to bounce your mix to audio first, then re-open Logic in 32 bit mode and bounce from there to your movie.
Anyway, hopefully that will solve your problem. If not, then you might have an issue with RAM which can be checked by doing an apple hardware test. I'm about to learn how to do that so can't advise as yet so hopefully someone else will have some further ideas or can fill in the blanks.
Hope this helps.
Nick