Welcome Guest! To enable all features please Login or Register.



Forum Jump  
Real memory vs. virtual memory
Last post Tue, Jan 18 2005 by seanmccoy, 14 replies.
Go to last post
Posted on Fri, Oct 29 2004 10:56
by strawinsky
Joined on Wed, Feb 05 2003, North America, Posts 403
I'm a relative novice in OSX.Can someone explain the meaning of virtual and real memory in the Activity Monitor? In relation to Logic, if I reach 4 gig of virtual RAM,when loading samples,should there be an alert telling me that's the limit,or will Logic simply crash?Or,is the 4 gig limit applicable to real memory?Thanks in advance.
Posted on Fri, Oct 29 2004 15:06
by stupid8track
Joined on Thu, Mar 13 2003, Posts 230

i can't pretend to actually be able to explain this, but i do know that the only 'limit' i've run into with logic is when Virtual Memory hits 3.5 GB - then i crash every time. hard crash, too. and this translates to somewhere around 3.2-3.3 GB of Real Memory.

i've asked around a lot and still don't quite understand the concept of Virtual Memory. i've been told it is really only a Virtual memory space that the application reserves for itself because it thinks it might be asked for that space. of course, i have no idea where this reservation takes place, if not in real memory.

anyway, the end result is that 3.5 GB VM is my ceiling with logic. perhaps evan or someone else can explain just what Virtual Memory is, where it lives, who it likes to hang out with on saturday's, etc...
Posted on Fri, Oct 29 2004 18:33
by strawinsky
Joined on Wed, Feb 05 2003, North America, Posts 403
Thanks,Stupid:that's consistent with my current experience/problem,which means that it isn't me,or my system,but a common phenomenon.Damned annoying:roll on Tiger.
Posted on Wed, Dec 15 2004 05:09
by dot
Joined on Tue, Jul 15 2003, Posts 91
stupid8track wrote:
perhaps evan or someone else can explain just what Virtual Memory is, where it lives, who it likes to hang out with on saturday's, etc...

That's a very useful question, stupid8track: where virtual memory lives, because it lives in your CPU(s). I mean, memory itself doesn't really know about virtual memory, just like your friend doesn't know where you drove from when you last showed up on his or her doorstep. Where you last parked your car doesn't affect your friend, and whether you use virtual memory or not doesn't affect your memory. Memory just sees data flying in and out of its addresses.

Virtual memory is kind of like a RAM memory VISA card for your computer's operating system. Virtual memory is just a way of presenting memory to applications, a way that isn't hard-wired 1:1 to physical limits. Just as you can pay off your VISA balance every month and avoid ever getting into debt, virtual memory CAN be used just to present the physical contents of memory at different addresses than are physically true. When your operating system overcommits memory, you are basically going into memory debt: the OS is telling the apps that you have quadruple (say) the amount of RAM than is physically installed in your machine. Your creditors in this analogy are your hard drives: when you try to put four gigs of data into virtual memory when you only have one gig of physical memory, the excess three gigs spills out onto your hard drive(s) in the form of IOU's. When someone tries to access memory contents that are present in virtual but not physical memory, something has to be kicked out of physical memory to disk in order to make room for what you want to bring back in from disk. It's basically a big check-kiting scheme realized in hardware.

Each CPU design has an "address space" limit -- the maximum amount of RAM it can address. If it supports virtual memory, then it also has a "virtual address space limit", and the virtual limit is usually larger than the physical limit. A 32-bit CPU typically addresses only on the order of 4 GB of RAM (that's 2^^32 bytes). So you're up against:

a) what the chip can address, physically and virtually, and
b) what the OS kernel reserves for its own use, and apps therefore cannot touch

I believe you two are discussing the virtual addressing limits of the G4, and the virtual addressing limit of the G5 when running in 32-bit mode.

Like the Windows crowd, EXS users have already worked out what the maximum amount of address space is left available to applications such as Logic and GS after OS X and Windows have carved out their share, which was determined by them when the OSes were coded. This also isn't something that is easy to re-code. In fact it is far easier never to budge it ever again after it is set. Most commercial vendors will say, "Wait for a chip with more address space (say 64-bit or 128-bit) rather than try to change the kernel's existing programmed share of virtual address space.

Roll on Tiger, indeed. It's a pity using my dual G5 in 32-bit mode this long after 64-bit operation became feasible.

As to your excellent question about who virtual memory likes to hang out with on Saturdays, I believe the correct answer is Linus Torvalds.

Logic knows how much virtual address space it is trying to "camp on" and reserve for itself, so it SHOULD be a good system citizen and warn you when you request loading so many samples that you "cross the line" which would crash the system. If the application doesn't make the effort to convey this, it is an omission on the part of Logic's programmers.

Take care,
Posted on Thu, Dec 16 2004 06:44
by stupid8track
Joined on Thu, Mar 13 2003, Posts 230

thanks for the great reply.

if you don't mind, there's one more thing regarding VM which i don't understand and that is Swap files.

i know a single Swap file is created when the OS is launched, but when and why are more created?

the reason i ask is that it seems to me that they are created well before actual memory is full. for example, my powerbook has 2 GB of RAM. i rebooted recently and am now 'using' 273 MB of actual RAM. there is 760 MB of inactive RAM, around 1 GB of 'free' RAM and a VM space of around 4.6 GB. and 1 swap file exists.

usually, a second swap file will get created when the 'actual' RAM gets to around 900 MB of space and the active+inactive is around 2 GB.

so what is the difference between VM and inactive RAM?

are Swap files just hard disk space that is assigned to be 'inactive' RAM when the 'inactive' RAM demand exceeds the amount of physical RAM?

is there any way to 'reduce' the size of inactive RAM demands to delay the creation of more Swap files?

thanks again for your previous reply.

Posted on Sat, Dec 18 2004 05:56
by dot
Joined on Tue, Jul 15 2003, Posts 91
Hi stupid (what an ironic name you have, grandma!)

As I mentioned earlier, VM doesn't imply overcommitment. An OS X example of that is if you prevent dynamic_pager from starting up (by commenting it out in your /etc/rc file), then VM is used only to assign pages of memory to different tasks in the system and share memory between them, but the contents of memory are never PAGED OUT to disk. I disabled dynamic_pager years ago on my PowerBook, and it's been running fine since. (It's running 10.2.8 now, but I just booted my dual G5 (10.3.5) to confirm that Panther will still boot without dynamic_pager running.)

When I run "top" with dynamic_pager disabled on my dual G5 (4GB RAM), it reports:

PhysMem: 174M wired, 104M active, 102M inactive, 381M used, 3.63G free
VM: 3.41G + 62.1M 14683(11) pageins, 0(0) pageouts

Top doesn't know overcommitment is disabled, and either way it reports VM as being the amount of pretend (virtual) RAM available in the system, only in this case virtual memory left over=physical memory left over.

Since the OS guys want the system to stay on its feet as well when booted with 128MB of RAM as when booted with 8GB of RAM, you can bet that a fair amount of work went into the heuristics for determining when to increase the amount of swap file space consumed. The worst thing for them is to have customers with tiny memory systems calling up complaining about crashes due to having optimized for best case, which is power-user customers like yourself with maxed-out RAM. It is a reasonable assumption for them that if you've got that much physical RAM, you can afford some wasted swap space on disk if it keeps the most people up & running. They can't optimize *for you*, but you can do things like disable page-outs or other memory tuning, though I have no experiencing tuning *enabled* swap-file space under OS X to share with you.

I guess you can see now that your reasonable question about the relationship between inactive RAM and reported VM isn't 1:1.

Did you notice the number of page-ins reported above by top, despite the number of page-outs being zero? That's because modern OSes don't bother reading code into memory from files and then swapping it out when they don't need it. Instead they use the COW (copy-on-write) approach: they memory-map the contents of the executables into memory, but don't populate the page frames with actual memory pages until each page frame is accessed. As your CPU moves from a populated page of memory to an unpopulated page frame, a page fault yanks the CPU away from anything it WAS doing to the page-in exception handler, which reads the missing page in from disk before returning to what it was trying to do before. This is all completely transparent -- you never see the interruptions that the CPU goes through. The faults slow you down, but then, any code that is part of your app that you never got around to executing, never needed to be read in from disk -- so it isn't.

EXS and other streaming software basically does the same thing: it memory-maps the contents of sample files into pages of virtual address space within your application (say EXS); and when you actually play a given sample at a given pitch, the processor traps and that page of data is sucked in from disk. It doesn't bother to open() and read() at this point -- that's all long been taken care of, it just pulls the corresponding blocks off the disk, directly into where it's needed in RAM.

It's fun stuff. Sorry if I over- or under-simplified it. I am way low on sleep, hope you are well. Thanks for your kind words about my last post.

Posted on Mon, Dec 20 2004 04:00
by stupid8track
Joined on Thu, Mar 13 2003, Posts 230

thanks again for another terrific post. really, my forum name is no accident Smile but you've made things clear enough that they even make sense to me. i really appreciate it.

one thing i would love to do is to disable dynamice paging on my system. i've maxed the RAM in both my powerbook and my G5, so i'd love to disable the dynamic paging.

if it isn't too much trouble, would you mind running through that quickly. i don't know how to do this:

(by commenting it out in your /etc/rc file)

do you mind stepping through how to do that?

also, does this have to be done after each reboot, or is it a do it once and you're done type of thing.

again, thanks for taking the time to explain this to an absolute layman. i really do appreciate it.

Posted on Mon, Dec 20 2004 04:55
by dot
Joined on Tue, Jul 15 2003, Posts 91
stupid8track wrote:
(by commenting it out in your /etc/rc file)

do you mind stepping through how to do that?

Hi stupid (I can't get used to calling you that!),

My dual G5's /etc/rc file looks like this at line #238:

dynamic_pager -F ${swapdir}/swapfile


As I said a year or so ago here on VSL forum when telling someone how to do this: I can't be responsible for what happens to your system if you do this. My laptop has 1 GB in it and my G5 4 GB in it, but I haven't run through the permutations of all the different OS releases and memory combinations and driver combinations to know if your system will boot if you do this. So only do it if you are comfortable with it after you read this, and if you have yourself sufficiently backed up, mostly because YOU CAN GET YOURSELF INTO A CATCH-22 SITUATION where you need to change the file back in order to boot, and you need to boot to change the file back. Don't say I didn't warn you, please.

Basic setup:
#1: open the Terminal application in Applications/Utilities
#2: sudo su -
(you'll get a nice and completely appropriate warning here.)
#3: type your administrator password to become root

(None of the rest will really work if you can't do the above.)

Backup your rc file:
#4: cd /etc
#5: cp rc rc.original (this copies file "rc" to "rc.original")

Edit your rc file. This is the hardest part to explain because I don't
have time to explain how to use vi, emacs, or another UNIX text editor.
#6: vi rc
#7: /dynamic_pager (this searches for the string "dynamic_pager")
#8: Type 'i' to start the Insert text command.
(You'll get a warning about the file being read-only. Proceed anyway.)
#9: Type '#' to insert a '#' comment character in front of dynamic_pager.
#10: Hit the [esc] key in the upper left corner of your keyboard to exit insert mode.
#11: Press the colon ':' key
#12: Type 'wq!' (the "write, quit, don't argue" command sequence)
#13: Hit the [return] key

If absolutely everything went well, you're now back at the UNIX command prompt and your rc file has been updated. You can check how it went by issuing the command:

grep 'dynamic_pager' rc

If you successfully commented out dynamic_pager, it'll look something like this:

#dynamic_pager -F ${swapdir}/swapfile

If there's no leading '#' character, you didn't comment out dynamic_pager, and it will still start up next time you boot.

Once you change this file, it will stay that way until you change it back. Every time you boot, it will do what it says in this file, so no, you don't have to keep commenting it out. That's why I gave you the big warning, really.

Anyway, if disaster ensues, log back in as root, cd to /etc, and copy your old rc file back over the modified version:

cp rc.original rc

At least this way you don't have to edit the file again -- no matter what you did to it, this should put its contents back to what they were before you started messing with it. If your system won't boot, you'l have to boot from another disk, login as root, and cd to /Volumes/Macintosh HD/etc first. (if your hard disk is named something other than Macintosh HD, substitute that instead under /Volumes.

Good luck. Please get in touch with your local UNIX guru before trying this, I'm going to be out this week, and it'll probably be January before I'm back on line here.

I admire you guys who can figure out how to use other people's software effectively. I am a babe in the woods at using Logic (and most other apps), would trade quite a lot to know half of what you know.

Safe and happy holidays all!
Posted on Mon, Dec 20 2004 06:06
by stupid8track
Joined on Thu, Mar 13 2003, Posts 230

again - thank you for taking the time to write such a clear, helpful email. i've got a cousin who is a unix guy and will be seeing him soon, so i'll hold off on this until then - but this post will definitely help tremendously.

and i understand completely that you can't make any guarantees as to the outcome/result/stability/function of my system if i choose to do this. no worries. i accept full responsibility for any and all tampering i do to my system (again - the forum name ain't an accident)

that said - i really appreciate your time and patience in explaining this. i've got 2 GB RAM in my powerbook, 8 in my G5 so i'd like to give it a run. i am sufficiently backed up at most times, but will be sure to back up again just before attempting anything.

and, unfortunately, half of what i know about logic wouldn't heat a turnover. but you're welcome to it. Smile

cheers again
Posted on Tue, Dec 21 2004 01:02
by dot
Joined on Tue, Jul 15 2003, Posts 91
I suppose I should say that it's not enough to be backed up -- you need a SECOND boot disk in order not to be rolling the dice.

I've never been locked out by disabling dynamic_pager, but back in OS X 10.1, I once accidentally typed /usr/lib instead of /usr/lib/somethingelse in a file rename (mv command) when trying to build an OS X to Intel cross-compiler on my TiBook, and renaming /usr/lib made my TiBook's OS X filesystem unbootable. The internal HD became useless until I ResEdit'ed it from an OS 9 boot CD.

Now THAT was stupid! And the solution was easy too, eh?

I suppose I should also warn you that Apple Support will categorically refuse to help you recover from anything you do to screw up your system while in Terminal. UNIX per se is unsupported -- they only support the Apple interface that runs on top of UNIX.

Bottom line is I don't want to lead you into doing ANYTHING to your system that prevents it from booting, even if all the data on your system remains intact. An unbootable system is way up there on the scale of world record time wasters.

Best wishes.
Posted on Wed, Dec 22 2004 01:18
by nick@madhatter
Joined on Fri, Jul 30 2004, Los Angeles, Posts 36
Dear Dot (and any other power users),
I consider myself a well-educated user...but still just a user, and not nearly as well educated as I'd like to be. I'd like to learn a lot more about Macs and the OS, but need a text that's at a good gradient to learn from. Can you recommend a book or website that could begin with? Let me know.

Posted on Thu, Jan 06 2005 15:09
by dot
Joined on Tue, Jul 15 2003, Posts 91
need a text that's at a good gradient to learn from. Can you recommend a book or website that could begin with?

Unfortunately, no. I haven't read any of the books that have come out since OS X shipped. I read the kernel mailing list for a couple of years, which is a bit of a firehose of disconnected information, even if you've done kernel-level coding for a decade or two, and it's just a bit much after a while. Too many operating systems to work on, and you know how many hours there are in a day.

The only good non-out-of-date text I've come across on this subject in the last several years is one on Solaris internals, and that's not the OS you care about.

Sorry for a poor, late response. Maybe someone else here can recommend a good one.
Posted on Mon, Jan 17 2005 07:03
by Zenzilla
Joined on Tue, Dec 28 2004, North Central Texas, Posts 7
A good OSX site to visit or get on mailing list is:


They have easy user & Unix stuff = enjoy!

Did someone say something about turnovers = Apple Turnovers?

Talking about music is like dancing about architecture ~ Frank Zappa
Posted on Tue, Jan 18 2005 06:21
by seanmccoy
Joined on Sat, Feb 28 2004, Central Point, OR, Posts 100
How about "OSX: The Missing Manual" by David Pogue? I bought it but have never yet looked at it. Anybody have any opinions on that book? It's over 500 pages.
Sean McCoy
Oregon Sound Recording
You cannot post new threads in this forum.
You cannot reply to threads in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.