Thanks for the replies. I tried lowering the buffer to zero but this didn't help at all. Two days ago I upgraded to VEPro 5 as I needed it anyway and am currently only using it on one local computer (Windows 7).
My audio interface is a Focusrite Scarlett 2i4 which isn't the best in terms of latency. It's around 20ms in general but this usually is OK for most VST instruments. The current latency of all instruments in VEPro feels like half/third of a second which is totally unplayable. Does the latency increase with more channels? I currently have 25 channels in VEPro.
If I use Vienna Instruments in Cubase itself without Ensemble Pro the latency is superb an playable. The settings in Ensemble are like this: Multiprocessing - 4 Threads, Plugin Latency Compensation Enabled. In the server interface the buffer is set to none but even there it adds another 5,2ms which should be OK!
I'm really not sure where this nasty latency comes from... I know that some latency comes from the CPU which in my case is an Intel i7-2600k with 3,4GHz.