Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Forum Jump  
Groundbreaking Workaround Discovered!!!
Last post Wed, Mar 24 2004 by evan evans, 161 replies.
Options
Go to last post
9 Pages<12345>»
Posted on Wed, Mar 10 2004 21:32
by evan evans
Joined on Tue, Jun 17 2003, Hollywood, CA, Posts 2058
stupid8track wrote:
for those of you coming from OS 9, with a G5/OS X/8GB RAM you can expect to load double the amount of instruments because logic can use twice as much actual RAM space for Virtual Memory as on a G4. perhaps, if evan can work some magic and get a 2nd logic running seamlessly for VSL, you'll be able to load and play 4x as many (exactly how much more RAM you can put in a G5 than a G4).

but that will be the limit. being seduced by numbers beyond that is only an affair of the heart. evan knows this, but he is currently busy turning the world inside out. and god bless him for it.
I stand by 350. I've been argued against before about my figures only to come out winning. For those of you not sure who to listen to, just stay open, and let's wait until we get to the end of this tunnel.

Evan Evans
Evan Evans
Film Scoring Academy
http://filmscoring.academy
Posted on Wed, Mar 10 2004 21:34
by stupid8track
Joined on Thu, Mar 13 2003, Posts 230
Quote:
So if you play back your 30 to 100 track piece which accesses 50 to 150 EXS instruments OUT OF 384 loaded, then after about 5 playbacks you will have all the resources aligned (ie: the necesary instrument's buffers will be mostly in RAM), and it will play without POP or CLICK.


so are you saying that every time you want one of those 300+ instruments to be loaded properly into RAM, you have to playback the track 5 times?

because i can't get 7 tracks to play simultaneously once disk swapping has been activated.

i'll try again right now, and try up to 20 playbacks to see if perhaps i just wasn't patient enough before.

again, i want this to be right, but in my experience it hasn't been.

cheers
Posted on Wed, Mar 10 2004 22:04
by evan evans
Joined on Tue, Jun 17 2003, Hollywood, CA, Posts 2058
Cyril wrote:
I have tried to sync 2 Logic task playing 1 track each Big Smile
One task is Master (output midi clock)
One task is slave (wait for Midi clock)
The problem is that the 2 Logic tasks are using the same pref files. Sad
So when you set one Logic to output Midi clock it may work for a while but you get conflicts after a few manipulations.
Also if your master task is a loop you lose the 1st note on the slaved Logic and you get "pshits"
Best
Cyril
Thanks for testing that Cyril. however that was wahat I did within the first 5 minutes of my discovery. I quickly outgrew it.

On to MIDI I/O with Logic in Loop Play.

Evan Evans
Evan Evans
Film Scoring Academy
http://filmscoring.academy
Posted on Thu, Mar 11 2004 00:20
by Carlos D
Joined on Tue, Oct 14 2003, Riverside, CA, USA, Posts 63
Ah, what a great thread.

Evan, good thought at the core here. I use this trick to get long StuffIt sessions done in parallel rather than waiting for one thing to get done, and then another. Hope this can all work out.... or get Logic Pro 7 out the door soon. Wink

A few thoughts:

---

1) Re 2GB per Thread (Nick)
It may be you misstated "per thread". It may be 2GB per thread. But a thread is not a process. I'd figure that each EXS instance takes at least 1 thread of its own (maybe more). Multiple threads live under the Logic process.

In Panther, the process size is still limited to 4GB. See;

http://www.osnews.com/story.php?news_id=4009

So even Logic 7 will be limited until 10.4, probably.

---

2) Evan et. al., have you thought of using Jack? It's in dev, but free. See:

http://www.versiontracker.com/dyn/moreinfo/macosx/22070

---

3) Caveat from the app developer in me:

While it may be a great work around, I would greatly caution all who attempt this on real work. I would definitely do this on a copy of a song. And keep all others safe. I'd also make backups of preference files before doing this.

As a programmer who has plenty of experience in programming multiple threads, I know that this stuff is incredibly difficult to fully debug. One of the things you have to watch for is simultaneous resource contention. If one instance touches a file (like prefs for example) while another tries to read it, you can get some very unexpected results. Unless Logic was designed with multiple simultaneous use in mind -- partially possible with the advent of fast user switching -- this could do weird things.

---

4) Hack EXS prefs?

As the point of this exercise is to work around the sample count allowed by the 3.5GB limit -- yes, having more EXS instances than 64 is very nice too, though I'd think it hard to hit 64 with the sample counts in VSL -- what if we were able to go down from the 8MB EXS buffer limit? If a G5 or fast G4 could keep up with only 4MB per head, you could double your samples, right?

My inclination would be to copy all the prefs, change the setting in EXS from fast to medium, and then examine the differences between pref files (unix 'diff' or Apple's File Merge developer tool). One may be able to decode the values and truly set your own. I'd think for the memory challenged, this may be a safer and more durable bet.

--
Enough for now. I look forward to hearing more progress!

Carlos
---
Hear my stuff:
http://music.altamar.dynalias.org/
just posted my music and soliciting comment --2/2004!
listen at: http://music.altamar.dynalias.org/
Posted on Thu, Mar 11 2004 02:19
by stupid8track
Joined on Thu, Mar 13 2003, Posts 230
Quote:
But OSX will self prioritze the resources it needs. So if you play back your 30 to 100 track piece which accesses 50 to 150 EXS instruments OUT OF 384 loaded, then after about 5 playbacks you will have all the resources aligned (ie: the necesary instrument's buffers will be mostly in RAM), and it will play without POP or CLICK


ok, evan, i removed 2 GB of RAM and fired up my previous song with 50 simultaneous parts going (song with 3.4 GB VM used, 300 MB Real Memory) and tried repeatedly to play it back without luck. i started playback 70 times but never got beyond the 5th bar. so i gave up.

so then i opened my version of the song which wouldn't play a 7th track (same song, only 7 midi parts) and i kept trying to play it back and lo and behold, after the 10th time, it played. you are correct about OSX loading the needed files into RAM.

i was then able to add parts on the remaining tracks, up to 50. which was really great. so i saved the song and reopened it - and couldn't get the thing to playback, even after another 70 attempts. close, reboot, reopen - can't get to play after 70 attempts.

so i removed 12 instruments, leaving 38 (song now 2.88 GB VM, 636 MB Real Mem) and saved as a new song. reopened it and was able to play it back after 25 starts. closed and reopened and it took 43 starts, but i could play it back eventually. closed and reopened, and it took 33 starts.

for the record, all of these songs played back on the first attempt with 4 GB of RAM in the machine. so while i might be inclined to believe your quote above regarding composition - that 5 playbacks will set memory for need - i definitely don't when it comes time to reopen the song and get to work.

there were, of course, several mystery crashes when working on the songs with only 2 GB, too. one when saving. one when adding an instrument (it wasn't a VM issue as the song had 750MB of VM space left and the file was only adding around 80MB). one when screen changing. another that i can't remember the cause.

so i still hold that once you start swapping files, you lose more than you gain.

but if you can get even a second logic to run as merely a giant sampler - giving users another 64 instruments and 3+ GB of sample load before swapping - then 8 GB of RAM might be useful.

and that would mean something big, i think. IMO, 300 instruments spread across 5 logic instances - even if it is possible - will exist as an experiment, not as a real world workflow. disk swapping, especially at that level, is still not desirable.

at least not by me. but keep up the good work - i'm learning something daily again.

as always - best of luck

by the way - is it possible for you to check the VSIZE and RSIZE numbers for your 40 instrument song in TOP? i'd be curious how much disk swapping is really happening there - and how many false starts it takes to play the thing back once you reopen it.

also - have you checked with Oink about environment stuff? he can really save you from anguish and false starts.
Posted on Thu, Mar 11 2004 06:37
by christianobermaier
Joined on Fri, Mar 05 2004, germany, Posts 68
>can't get to play after 70 attempts.

Not really encouraging when you're composing under the gun with a client breathing down your neck... ;o)

I still wonder how the Environment latency issued could be addressed, since in the end you do not really gain that much when you can't play more than one EXS per Logic instance in sync.

Christian


[URL=http://uk.geocities.com/christianobermaier/Gearlist.htm]gear list [/URL] studio pics show reel
Posted on Thu, Mar 11 2004 07:22
by jbm
Joined on Fri, Jan 16 2004, Posts 1159
8track,

I'm just wondering: are you running Panther?

It seems vm usage has been changed quite dramatically. I know that when I was using Jaguar, my system basically never paged out with 1 GB physical RAM. I wasn't loading massive amounts of samples, not enough to "technically" require a pageout. However, in Panther it starts paging out _long_ before I load enough to even touch the 1 GB physical memory. So, I did some research, and it looks as though Panther now dynamically pages-out processes continually, by priority. Maybe it always did, but I certainly didn't observe it this dramatically before.
It follows, then, that perhaps evan is seeing the dynamic paging out of ram, thus allowing all sample heads to be prioritized and kept in physical memory...? yes, no, possible?
Posted on Thu, Mar 11 2004 08:01
by evan evans
Joined on Tue, Jun 17 2003, Hollywood, CA, Posts 2058
Thanks Stupid8Track,

yeah I have my mind wrapped pretty well around all this stuff and that's why I understand that with at least 2 Logics and 8GB Ram 250 Instuments is a reality.

But onyl a reality if we can figureout how to patch into all 64 instances onteh other Logics. Of course it's a piece of cake but I am not knowledeable enough about Environment to make it happen. Then I could test for latency, perhaps with I/O trick. I am so ready to do this, but haven't yet posted to OINK.

The self prioritization gets geometrically worse with less RAM, not linearly. So popping out some RAM and then extrapolating those results out will not get you the correct conclusion needed. RAM is exponentially addressed, and more RAM requires very little extra effort to use, but less RAM requires lots of effort to use. Exponentially so.

Glad you are seeing some of those thigns which I am using as my power tools. Another is my "workaround" as far as sample limit is concerned. That was an incredible "workaround" that I wound up "working around".

Smile

Evan Evans
Evan Evans
Film Scoring Academy
http://filmscoring.academy
Posted on Thu, Mar 11 2004 08:04
by evan evans
Joined on Tue, Jun 17 2003, Hollywood, CA, Posts 2058
Stupid8track,

Also, lookup "renice" on the internet. I think you'll get that song to play waaay before 70 attempts. And cehckout your vnodes, etc. These hurdles are minor.

And as far as Sample Instrument latency, this can be compensated for if need be. Yes, alittle less than professional, but maybe just fine.

Evan Evans
Evan Evans
Film Scoring Academy
http://filmscoring.academy
Posted on Thu, Mar 11 2004 08:06
by evan evans
Joined on Tue, Jun 17 2003, Hollywood, CA, Posts 2058
jbm wrote:
It follows, then, that perhaps evan is seeing the dynamic paging out of ram, thus allowing all sample heads to be prioritized and kept in physical memory...? yes, no, possible?
eeeeeh .... in a sense. Those things happen, but not quite like and becuase of what you said. But certainly this is the effect of the Panther upgrade. A new feature was implemented which gives apps more vnodes and also the VM manager was improved.

Evan Evans
Evan Evans
Film Scoring Academy
http://filmscoring.academy
Posted on Thu, Mar 11 2004 08:55
by christianobermaier
Joined on Fri, Mar 05 2004, germany, Posts 68
> as far as Sample Instrument latency, this can be compensated for if need be. Yes, alittle less than professional, but maybe just fine.

Given that you manage to pipe all MIDI streams into the respective Logic instances, you can patch 4x16 channels per Logic through Channel Splitters to 64 EXSs. Since all EXSs would suffer from the environment latency, you would need to resort to mouse input only. No realtime playing from the keyboard anymore.

Christian

[URL=http://uk.geocities.com/christianobermaier/Gearlist.htm]gear list [/URL] studio pics show reel
Posted on Thu, Mar 11 2004 11:02
by evan evans
Joined on Tue, Jun 17 2003, Hollywood, CA, Posts 2058
Anyone heard or know why environment latency exists? Was it like that in OS9? I mean it is unuseable.

Evan Evans
Evan Evans
Film Scoring Academy
http://filmscoring.academy
Posted on Thu, Mar 11 2004 13:10
by christianobermaier
Joined on Fri, Mar 05 2004, germany, Posts 68
Yeah, it has always been like this. Many tears have been shed over this issue. It is essentially this what keeps folks from using Logic as an absolute killer VSTi host for realtime on-stage work.

As i mentioned, the IO trick works, but you're not going to get serious mileage out of this.

Christian

[URL=http://uk.geocities.com/christianobermaier/Gearlist.htm]gear list [/URL] studio pics show reel
Posted on Thu, Mar 11 2004 16:19
by Aubrey Robertson
Joined on Mon, Oct 27 2003, Posts 116
I’m out of the office this week, because of a new baby, but thought I would share what I have been working on. I have one dual 2.0 G5 with Logic and Pro Tools installed, and 8g of ram. What I found was that the two apps running together eat up too much cpu to really max out either. With Logic alone I can get 30+ instruments streaming, but with PT running too, that drops to 8-10. About latency, using wormhole or Jack the latency streaming audio to Pro Tools is unusable. Also, you get occasional pops and glitches because of clock issues between Logic and PT. Using an RME interface on the same machine fixed the latency issues and the audio glitches, but still no better performance as far as quantity of instruments.

I decided to add another G5 and had just begun testing last week. Here’s what I know so far. Latency from PT to Logic is fine using the RME interface from the second Mac to stream into the PT hardware. I am using channel splitters, bypassed I/O plugs and loop mode as is detailed in other threads. However, using wormhole over Ethernet still has clock and latency issues. I have my interfaces word clocked together, and have tried about everything, but the audio swap between the two macs is just not stable with the software. This may be a problem with the FXpansion vst to rta adapter. The next thing I was going to try was streaming from Logic to Logic on the two macs to see if the clock issue could be resolved. I also wondered if I would be able to run 6.3.2 and 6.3.3 together on the same mac, but I hadn’t considered the brilliantly simple thought of running two versions of the same logic.

I am very curious if anyone is able to use wormhole or jack to stream from one logic to the other on the same mac. I am excited by evan’s enthusiasm, but also tempered by stupid8tracks real world perspective. Loading many instruments is the easy part. Actually streaming many instruments is what eats up resources. CPU usage increases dramatically when so many instruments are used at once. I am encouraged by the prospects, but am cautiously optimistic. The real world performance of a G5 is great, but it is not quite up to much of the hype about its supercomputing abilities.

Aubrey
G5 dual 2.0 OS 10.3.2|Logic 6.3.3|VSL Pro Edition
Posted on Thu, Mar 11 2004 16:32
by stupid8track
Joined on Thu, Mar 13 2003, Posts 230
jbm

yes, i'm on panther (10.3.2). i don't think i've even tried jaguar for heavy EXS VSL use because, at the time, there were all those problems playing multiple instruments and then the open file limits.

evan

Quote:
The self prioritization gets geometrically worse with less RAM, not linearly. So popping out some RAM and then extrapolating those results out will not get you the correct conclusion needed. RAM is exponentially addressed, and more RAM requires very little extra effort to use, but less RAM requires lots of effort to use. Exponentially so.


so it stands to reason that self prioritization gets geometrically worse with greater swapping. so we are saying the same thing - swapping is bad for streaming. can be used in emergency/desperation, but with limits.

in the end, with the performance hit i see with swapping and the speed with which the samplemerged instruments load, i think most people will choose not to swap just to have everything pre-loaded.

Quote:
250 Instuments is a reality.


i guess it depends on the instruments, really. in the end, there will be a sample load limit, which can be divided up into instruments as desired by the user. in that respect, what might be 100 to me, could be 250 to you. with the stuff i've been doing, i run out of RAM at 50 or so instruments - but you might consider them to be more than 50 instruments as some are ALL, some are dynamic multi-layered, etc.

Quote:
But onyl a reality if we can figureout how to patch into all 64 instances onteh other Logics. Of course it's a piece of cake but I am not knowledeable enough about Environment to make it happen. Then I could test for latency, perhaps with I/O trick. I am so ready to do this, but haven't yet posted to OINK.


not sure it is a piece of cake - it appears to be anything but. and while i think Oink is an environment whiz, so is christian. and what christian says about no one being able to make logic work as a simple VSTi host is correct, from what i've seen. but if anyone can help you with this, it would be christian and Oink. if they can't, beware.

Quote:
Glad you are seeing some of those thigns which I am using as my power tools. Another is my "workaround" as far as sample limit is concerned. That was an incredible "workaround" that I wound up "working around"


yes, that was quite a good work around - i don't mean to diminish your abilities or your history. both are well established (though that comment about a G5 with EXS/VSL equalling 20+ gigastudios was "just a bit outside") i just don't think this one is going to work out - for a series of reasons. swapping being one of them.

Quote:
And cehckout your vnodes, etc. These hurdles are minor.


for the record, when i switched to 2 GB RAM, i reset my maxvnodes, maxfiles, and maxfilesperproc settings to 384000, 128000, 96000 respectively upon each reboot. i don't think my trouble was the vnodes.

anyway, i wish you the best of luck and, if anyone can get it done, you can. i just don't think anyone can get it done. IMO, our best hope is still 64 bit OS and 64 bit Logic. until then, 64 instruments and 3.2 GB of sample load is it. and a G5 with 4GB RAM eliminates swapping.
Posted on Thu, Mar 11 2004 18:51
by evan evans
Joined on Tue, Jun 17 2003, Hollywood, CA, Posts 2058
stupid8track wrote:
...so it stands to reason that self prioritization gets geometrically worse with greater swapping. so we are saying the same thing - swapping is bad for streaming. can be used in emergency/desperation, but with limits.
Oh jeez, NO. I guess I didn't explain it right. As you get into deeper bit depths of RAM the VM algorithm improves greatly. Twice as much RAM will yield four times better CPU performance, etc.
stupid8track wrote:
... for the record, when i switched to 2 GB RAM, i reset my maxvnodes, maxfiles, and maxfilesperproc settings to 384000, 128000, 96000 respectively upon each reboot. i don't think my trouble was the vnodes....
Do not forget about the simple one word i used above:

renice

Check it out. It'll get things streaming before 70 attempts.

Smile

Evan Evans
Evan Evans
Film Scoring Academy
http://filmscoring.academy
Posted on Fri, Mar 12 2004 03:43
by stupid8track
Joined on Thu, Mar 13 2003, Posts 230
well evan, i am a VERY happy man because you are absolutely correct about OS X and Virtual Memory and disk swapping. i mean, 100 percent. i couldn't have been more wrong. i have finally seen the light.

quickly - i pulled out 2 GB of RAM and tried again to open and play that 50 instrument song and couldn't after 70 attempts to start it. so i rebooted, reset all the maxfiles, launched logic and reset the renice setting to -18, opened the song and still couldn't get it to play after 70 attempts to start it. i have no idea why.

anyway, i put the other 2 GB back in the machine and rebooted and opened the 50 instrument song - played fine, of course. so then i closed it and created a new song with non-samplemerged legato instruments (coming from a different drive to be sure i didn't repeat any). loaded 12 - the 13h gave me the too many open files error. played parts on all 12. then loaded the 50 instrument song.

and after 7 starts, it played. while the 12 instrument song was playing, too. so i opened a 3rd logic instance and added one Violin All instrument before i ran out of Virtual Memory. but i put a part on that and it played too. i could hear them all at the same time as they were all using the same audio driver.

rebooted the machine, reloaded the songs, and all played fine (the 50 instrument song required 7 starts, but so what). in fact, Virtual Memory was more efficient this time and i added 2 more instruments (for a total of 65) and all played.

so you are absolutely correct. i have no idea why i wasn't able to achieve this with only 2 GB of RAM - undoubtedly user error of some kind, or maybe a RAM thing. i don't know. but it works. thank you for your patience and assistance.

now for the REALLY good news (at least for me):

the logic control works as a global controller. using it, when i hit play - ALL logic instances start and stop together, synchornized from the same point in the song. when i use the scrub wheel, i scrub through all the songs together - when i lower the master fader, the volume in all logics lowers, etc.

and since all instances are using the same audio driver and i can hear them all playing back at once, i can just select any track in the arrange window of any of the logics and hit record in the transport (not on the LC, unless i want to record on that channel in all the logics at once) and compose away.

as it turns out, the extra logic's really are just additional AI channels for composing. no environment latency or audio clocking issues or sync issues to worry about.

so now i can set up separate Logic songs, each with a separate VSL section, and when i want to use one, just open a new logic instance and open the song there. then just switch back and forth between arrange windows according to what i want to use. if i want to hear just the violins, i hit play with the mouse on that sections transport. if i want to hear everything and/or see it to picture, i hit play on the logic control. screensets and keycommands will help for navigation.

and, of course, each section can have its own effects, though you might be able to work something magical with wormhole or any of the other audio utilities. me, i'm happy to just compose without trouble and then bounce when done.

anyway, thank you again for all your (continuing) hard work. and thanks for your patience with my ignorance. i am very happy that you were right.

buying more RAM now.

cheers
Posted on Fri, Mar 12 2004 03:46
by stupid8track
Joined on Thu, Mar 13 2003, Posts 230
i mean, truly, thank you.
Posted on Fri, Mar 12 2004 04:50
by stupid8track
Joined on Thu, Mar 13 2003, Posts 230
and, for the record, i didn't run renice on my system with 4 GBs - nor did i change any of the max (files, etc).

and i just rebooted, retested, and all still works well. i think the smart thing to do is just create many duplicate Logics and keep them all in your doc, then create a bunch of VSL template songs based on workflow.

if you want to have a bunch of songs which are sections and then one or two which are general purpose, great (they might need to have a performance on each track at boot to make it easier to help OS X, i don't know). then, whenever you need one, just click open a second logic in the dock and open the desired song/section from within it and let it load in the baçkground, while working on your song.

and don't forget about Expose in Panther - makes navigating between logic instances much easier.

again, evan, thanks for making your point over and over again. apparently, it is just what i needed to hear. Smile

cheers
Posted on Fri, Mar 12 2004 16:19
by Craig Sharmat
Joined on Wed, Oct 16 2002, Woodland Hills, Posts 416
Wow!!

this is great. Maybe I am missing something here but it seems to get this to work you have restart your song 7 times or am i missing something? I understand being able to use all this memory now on one G5 is a great thing ....but restarting 7X seems to be for a scoring musician a pain in the butt. I guess you could save this as your template for large arrangements (I hope this would work) but i wonder if there is a workaround for the 7 restarts.
9 Pages<12345>»
You cannot post new threads in this forum.
You cannot reply to threads in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.