strawinsky

i can't pretend to actually be able to explain this, but i do know that the only 'limit' i've run into with logic is when Virtual Memory hits 3.5 GB - then i crash every time. hard crash, too. and this translates to somewhere around 3.2-3.3 GB of Real Memory.

i've asked around a lot and still don't quite understand the concept of Virtual Memory. i've been told it is really only a Virtual memory space that the application reserves for itself because it thinks it might be asked for that space. of course, i have no idea where this reservation takes place, if not in real memory.

anyway, the end result is that 3.5 GB VM is my ceiling with logic. perhaps evan or someone else can explain just what Virtual Memory is, where it lives, who it likes to hang out with on saturday's, etc...

That's a very useful question, stupid8track: where virtual memory lives, because it lives in your CPU(s). I mean, memory itself doesn't really know about virtual memory, just like your friend doesn't know where you drove from when you last showed up on his or her doorstep. Where you last parked your car doesn't affect your friend, and whether you use virtual memory or not doesn't affect your memory. Memory just sees data flying in and out of its addresses.

Virtual memory is kind of like a RAM memory VISA card for your computer's operating system. Virtual memory is just a way of presenting memory to applications, a way that isn't hard-wired 1:1 to physical limits. Just as you can pay off your VISA balance every month and avoid ever getting into debt, virtual memory CAN be used just to present the physical contents of memory at different addresses than are physically true. When your operating system overcommits memory, you are basically going into memory debt: the OS is telling the apps that you have quadruple (say) the amount of RAM than is physically installed in your machine. Your creditors in this analogy are your hard drives: when you try to put four gigs of data into virtual memory when you only have one gig of physical memory, the excess three gigs spills out onto your hard drive(s) in the form of IOU's. When someone tries to access memory contents that are present in virtual but not physical memory, something has to be kicked out of physical memory to disk in order to make room for what you want to bring back in from disk. It's basically a big check-kiting scheme realized in hardware.

Each CPU design has an "address space" limit -- the maximum amount of RAM it can address. If it supports virtual memory, then it also has a "virtual address space limit", and the virtual limit is usually larger than the physical limit. A 32-bit CPU typically addresses only on the order of 4 GB of RAM (that's 2^^32 bytes). So you're up against:

a) what the chip can address, physically and virtually, and
b) what the OS kernel reserves for its own use, and apps therefore cannot touch

I believe you two are discussing the virtual addressing limits of the G4, and the virtual addressing limit of the G5 when running in 32-bit mode.

Like the Windows crowd, EXS users have already worked out what the maximum amount of address space is left available to applications such as Logic and GS after OS X and Windows have carved out their share, which was determined by them when the OSes were coded. This also isn't something that is easy to re-code. In fact it is far easier never to budge it ever again after it is set. Most commercial vendors will say, "Wait for a chip with more address space (say 64-bit or 128-bit) rather than try to change the kernel's existing programmed share of virtual address space.

Roll on Tiger, indeed. It's a pity using my dual G5 in 32-bit mode this long after 64-bit operation became feasible.

As to your excellent question about who virtual memory likes to hang out with on Saturdays, I believe the correct answer is Linus Torvalds.

Logic knows how much virtual address space it is trying to "camp on" and reserve for itself, so it SHOULD be a good system citizen and warn you when you request loading so many samples that you "cross the line" which would crash the system. If the application doesn't make the effort to convey this, it is an omission on the part of Logic's programmers.

Take care,
dot

Hi stupid (what an ironic name you have, grandma!)

As I mentioned earlier, VM doesn't imply overcommitment. An OS X example of that is if you prevent dynamic_pager from starting up (by commenting it out in your /etc/rc file), then VM is used only to assign pages of memory to different tasks in the system and share memory between them, but the contents of memory are never PAGED OUT to disk. I disabled dynamic_pager years ago on my PowerBook, and it's been running fine since. (It's running 10.2.8 now, but I just booted my dual G5 (10.3.5) to confirm that Panther will still boot without dynamic_pager running.)

When I run "top" with dynamic_pager disabled on my dual G5 (4GB RAM), it reports:

PhysMem: 174M wired, 104M active, 102M inactive, 381M used, 3.63G free
VM: 3.41G + 62.1M 14683(11) pageins, 0(0) pageouts

Top doesn't know overcommitment is disabled, and either way it reports VM as being the amount of pretend (virtual) RAM available in the system, only in this case virtual memory left over=physical memory left over.

Since the OS guys want the system to stay on its feet as well when booted with 128MB of RAM as when booted with 8GB of RAM, you can bet that a fair amount of work went into the heuristics for determining when to increase the amount of swap file space consumed. The worst thing for them is to have customers with tiny memory systems calling up complaining about crashes due to having optimized for best case, which is power-user customers like yourself with maxed-out RAM. It is a reasonable assumption for them that if you've got that much physical RAM, you can afford some wasted swap space on disk if it keeps the most people up & running. They can't optimize *for you*, but you can do things like disable page-outs or other memory tuning, though I have no experiencing tuning *enabled* swap-file space under OS X to share with you.

I guess you can see now that your reasonable question about the relationship between inactive RAM and reported VM isn't 1:1.

Did you notice the number of page-ins reported above by top, despite the number of page-outs being zero? That's because modern OSes don't bother reading code into memory from files and then swapping it out when they don't need it. Instead they use the COW (copy-on-write) approach: they memory-map the contents of the executables into memory, but don't populate the page frames with actual memory pages until each page frame is accessed. As your CPU moves from a populated page of memory to an unpopulated page frame, a page fault yanks the CPU away from anything it WAS doing to the page-in exception handler, which reads the missing page in from disk before returning to what it was trying to do before. This is all completely transparent -- you never see the interruptions that the CPU goes through. The faults slow you down, but then, any code that is part of your app that you never got around to executing, never needed to be read in from disk -- so it isn't.

EXS and other streaming software basically does the same thing: it memory-maps the contents of sample files into pages of virtual address space within your application (say EXS); and when you actually play a given sample at a given pitch, the processor traps and that page of data is sucked in from disk. It doesn't bother to open() and read() at this point -- that's all long been taken care of, it just pulls the corresponding blocks off the disk, directly into where it's needed in RAM.

It's fun stuff. Sorry if I over- or under-simplified it. I am way low on sleep, hope you are well. Thanks for your kind words about my last post.

dot

Hi stupid (I can't get used to calling you that!),

My dual G5's /etc/rc file looks like this at line #238:

dynamic_pager -F ${swapdir}/swapfile

appprofiledir=/private/var/vm/app_profile
...

As I said a year or so ago here on VSL forum when telling someone how to do this: I can't be responsible for what happens to your system if you do this. My laptop has 1 GB in it and my G5 4 GB in it, but I haven't run through the permutations of all the different OS releases and memory combinations and driver combinations to know if your system will boot if you do this. So only do it if you are comfortable with it after you read this, and if you have yourself sufficiently backed up, mostly because YOU CAN GET YOURSELF INTO A CATCH-22 SITUATION where you need to change the file back in order to boot, and you need to boot to change the file back. Don't say I didn't warn you, please.

Basic setup:
#1: open the Terminal application in Applications/Utilities
#2: sudo su -
(you'll get a nice and completely appropriate warning here.)
#3: type your administrator password to become root

(None of the rest will really work if you can't do the above.)

Backup your rc file:
#4: cd /etc
#5: cp rc rc.original (this copies file "rc" to "rc.original")

Edit your rc file. This is the hardest part to explain because I don't
have time to explain how to use vi, emacs, or another UNIX text editor.
#6: vi rc
#7: /dynamic_pager (this searches for the string "dynamic_pager")
#8: Type 'i' to start the Insert text command.
(You'll get a warning about the file being read-only. Proceed anyway.)
#9: Type '#' to insert a '#' comment character in front of dynamic_pager.
#10: Hit the [esc] key in the upper left corner of your keyboard to exit insert mode.
#11: Press the colon ':' key
#12: Type 'wq!' (the "write, quit, don't argue" command sequence)
#13: Hit the [return] key

If absolutely everything went well, you're now back at the UNIX command prompt and your rc file has been updated. You can check how it went by issuing the command:

grep 'dynamic_pager' rc

If you successfully commented out dynamic_pager, it'll look something like this:

#dynamic_pager -F ${swapdir}/swapfile

If there's no leading '#' character, you didn't comment out dynamic_pager, and it will still start up next time you boot.

Once you change this file, it will stay that way until you change it back. Every time you boot, it will do what it says in this file, so no, you don't have to keep commenting it out. That's why I gave you the big warning, really.

Anyway, if disaster ensues, log back in as root, cd to /etc, and copy your old rc file back over the modified version:

cp rc.original rc

At least this way you don't have to edit the file again -- no matter what you did to it, this should put its contents back to what they were before you started messing with it. If your system won't boot, you'l have to boot from another disk, login as root, and cd to /Volumes/Macintosh HD/etc first. (if your hard disk is named something other than Macintosh HD, substitute that instead under /Volumes.

Good luck. Please get in touch with your local UNIX guru before trying this, I'm going to be out this week, and it'll probably be January before I'm back on line here.

I admire you guys who can figure out how to use other people's software effectively. I am a babe in the woods at using Logic (and most other apps), would trade quite a lot to know half of what you know.

Safe and happy holidays all!
dot

I suppose I should say that it's not enough to be backed up -- you need a SECOND boot disk in order not to be rolling the dice.

I've never been locked out by disabling dynamic_pager, but back in OS X 10.1, I once accidentally typed /usr/lib instead of /usr/lib/somethingelse in a file rename (mv command) when trying to build an OS X to Intel cross-compiler on my TiBook, and renaming /usr/lib made my TiBook's OS X filesystem unbootable. The internal HD became useless until I ResEdit'ed it from an OS 9 boot CD.

Now THAT was stupid! And the solution was easy too, eh?

I suppose I should also warn you that Apple Support will categorically refuse to help you recover from anything you do to screw up your system while in Terminal. UNIX per se is unsupported -- they only support the Apple interface that runs on top of UNIX.

Bottom line is I don't want to lead you into doing ANYTHING to your system that prevents it from booting, even if all the data on your system remains intact. An unbootable system is way up there on the scale of world record time wasters.

Best wishes.
dot

Unfortunately, no. I haven't read any of the books that have come out since OS X shipped. I read the kernel mailing list for a couple of years, which is a bit of a firehose of disconnected information, even if you've done kernel-level coding for a decade or two, and it's just a bit much after a while. Too many operating systems to work on, and you know how many hours there are in a day.

The only good non-out-of-date text I've come across on this subject in the last several years is one on Solaris internals, and that's not the OS you care about.

Sorry for a poor, late response. Maybe someone else here can recommend a good one.
dot

How about "OSX: The Missing Manual" by David Pogue? I bought it but have never yet looked at it. Anybody have any opinions on that book? It's over 500 pages.