JWColby
jwcolby at colbyconsulting.com
Thu Aug 31 13:48:37 CDT 2006
Jim, That "in a nutshell" is a very nice explanation. I understand (at a conceptual level) all of this. I used to be in hardware long ago (back when we were swapping memory up above 1 MEGABYTE), and knew the processor instructions that did this stuff etc. I left the hardware side really just about the '386 era. I never really learned the "ring" stuff and protection beyond generalities as it wasn't really added until the '386. What I am trying to discover now, again in generalities, is what really happens specifically in XP32 (cause that's what I am running now) if you add more memory. I assume that memory up to the point where the amount runs into the issue of memory mapped onto hardware (video cards and the like) will just be added to the available pool and used as needed. But (for example) if 4 gig is in a machine and the OS maps .75 gig to "hardware", is the .75 gig of real physical ram just sitting there unused? In the end it may not make any difference in terms of the decision to buy 3 or 4 gig. For example putting a single stick into a MB for an AMD64 can cause performance issues so perhaps you buy two just to get the performance increase even if part of it just sits there. Just kinda "nice to know" on that one. What happens with the swap file and physical ram is a different story though. As I discovered, I cannot even make the swap file larger than 4 gig which implies that the OS decides that since it can only ever use 4 gig, that is the max it can swap out anyway. But that is a rather short sighted way of viewing things. Let's say for the sake of argument that you are doing some video editing and that app asks for 2 gig. You also run some other memory hungry something such as playing a modern game, and it asks for 2 gig. In theory each app should be able to ask for (but not necessarily receive) 2 gig of real physical memory. Just to push the envelope let's say that you load FOUR different memory hog programs and each asks for 2 gig of ram. That is 8 gig asked for. Obviously only 4 gig is physically present, and apparently to XP only 4 gig is truly available, but in theory the swap file should be able to go to 8, 16 or even 32 gig to handle the overflow that has to be swapped in and out. Limiting the swap file to 4 gig seems to be a pretty serious and rather arbitrary limitation here. We have been swapping memory since the dark ages, way before I ever got into the business. It seems that an OS as recent as XP would have opened things up a bit. John W. Colby Colby Consulting www.ColbyConsulting.com -----Original Message----- From: accessd-bounces at databaseadvisors.com [mailto:accessd-bounces at databaseadvisors.com] On Behalf Of Jim Dettman Sent: Thursday, August 31, 2006 2:17 PM To: 'Access Developers discussion and problem solving' Subject: Re: [AccessD] Using 4 gbytes of RAM John, <<All of my machines are 64 bit (AMD) processors so theoretically they can use more than 4 gig but how?>> A. The OS must support it. B. Executables must be compiled for it. Where all the confusion comes in is in talking about address space vs memory. They are two separate things and physical memory is not utilized anything like your thinking. For a given processor / OS, there is a maximum amount of memory that can be addressed by any given process. This is called the virtual address space. The OS in order to run the process takes up some of that and is mapped into that virtual address space (it usually takes the upper half and there is only one copy mapped into each processes address space). To add to this, some of the space assigned for the process is also used for the OS. Some of that is swappable (can be moved to the swap file it needed) or resident (must always stay in physical memory). Now it gets a little complicated. Each process is given a working set, that is a number of memory pages it can use at any given time. In addition to that there is a page frame table. This is a list of pages that a process is using and they are mapped into the virtual address space. In other words a program *thinks* it can address 4GB of space, but until it wants it, it's not tracked. For each page table entry, the OS keeps track of the virtual address space that it's mapped to and if the page is in the current working set, on the modified page or free page lists, or if it has been paged out to disk. For the working sets, most OS's have a couple different settings that can be played with: Working set default. Working set quota. Working set Extent. Working set increment. Working set decrement. To keep track of physical memory, there are two other lists in the OS: Free page list Modified page list The free page list is use to keep track of physical pages that are available for use by any process. The modified list are physical pages that have been modified by a process, but have not yet been written out to the page file (I'll get to this in a minute). They also have some internal settings to control page faulting. When a process is created, it starts at the default working set size. As the process needs more memory (ie. uses a variable) the OS adds an entry to the page frame table, grabs a page of memory from the free page list, and assigns it to the working set. Once the processes working set is used up, the OS sees if it can be expanded. If so, it expands by the increment. It does this until it reaches the working set extent. If a process still needs more memory, it needs to give up a page in it's working set. The OS takes one of the working set pages, writes it to the modified page list if it's been changed or the free page list if it has not (it's written to the bottom of the list), then grabs a new page from the from the top of the free page list and puts it in the working set. This process continues on. If a process no longer needs the memory, it's released from the working set and put back on the free page list. In the background as part of the OS is a process called the swapper, which controls all this. It monitors the free page list. If not enough free memory is available, it takes the pages from the modified page list, writes them to the page file on disk, and then moves those pages to the bottom of the free page list. What happens if a process needs a page back that it gave up from it's working set? First the OS checks if it is on either the free or modified page lists. If so, this is called a soft fault and the OS can easily move the page back into the working set. If however it was written to the page file on disk (because the OS ran low on physical memory), it's a hard fault. Last, if the OS runs really low on memory, it can elect to swap out a process and it's entire working set. It will do this only if: 1. The modified page list has been flushed. 2. All processes working sets have been trimmed back to the working set quota. Swapping is expensive and done only as a last resort. A process will also swap out if the overall number or processes allowed to be memory resident at one time has been exceeded. I haven't explained everything, like balance set slots, etc. But that's the way virtual memory works in a nutshell. It varies by the OS, but that's been the model used by VMS, NT, XP, etc. Jim.