I've been trying to patch ntoskrnl.exe on Windows Home Server 2011 to increase the memory limit from 8GB to 16GB (or simply remove it altogether). I've been operating on the presumption that I can patch it in the same way Vista 32-bit was patched to remove the 4GB limit ("http://www.geoffchappell.com/notes/windows/license/memory.htm").

After spending the past few days poking through disassembly, I'm at a bit of a loss. Even with symbols, there doesn't seem to be a Unicode string for Kernel-WindowsMaxMemAllowedx64 anywhere. Also, I can't find MxMemoryLicense anywhere, even after loading symbols, but if I open the .pdb file in a hex editor, I it does contain MxMemoryLicense.

I'm hoping someone here is more informed on how the limit is applied in 64-bit Windows and can nudge me in the right direction. I'm not really looking for someone to give the answer outright as I'm hoping to learn something new in the process.

Also, in case it helps, WHS2011 SP1 uses the same ntoskrnl.exe as Windows 7 x64 SP1 and Windows 2008 R2 SP1, so the limit would be loaded in the same way in those systems.

I think I've found the source of my problem. For some reason IDA doesn't load the entire NTOSKRNL.EXE. When I open it in a hex editor, I found a Unicode string "Kernel-WindowsMaxMemAllowedx64" at $4DBFB0. However, in IDA the file ends at $4BBFF0.

Why doesn't IDA load the entire file? I must be missing something.

Not sure if you check this, but have you made progress. I've been poking around and the ntoskrnl.exe that I'm looking at doesn't match Geoff Chappell's article. I even found a dup2 file that I tried to run, but it could not find the proper strings to modify. I upgraded my server to 16gb and now it seems wasted. I was trying to make a 8gb ramdisk, but couldn't get it to allocate properly since the memory is listed as hardware reserved.