Posted on behalf of Jordan Hargrave, Linux Software Engineer
Several Dell PowerEdge systems now support the capability of using 1TB RAM. On systems with AMD processors, you may need to use certain command line options to make sure that your system functions correctly. The default IOMMU only supports 40 bit physical addresses, which causes problems on systems which have remapped the memory hole above 1TB.
The IOMMU is used to remap system buffers to addresses that a device supports. Some devices cannot do I/O above the 4Gb limit and must use bounce buffers or other method to be able to perform I/O on system buffers which can be located anywhere in memory. On an AMD platform there are three different IOMMUs which can be used, AMD-GART (default), Software and AMD-Vi. The default GART only supports 40-bits of physical address... if a system buffer is located above this limit then the DMA I/O will go to the wrong location in memory, causing corruption or system hangs. Therefore a different IOMMU must be used, or the system must limit the amount of memory available.
The software IOMMU creates a bounce buffer below the 4Gb limit. This IOMMU performs basic memory copies to/from the bounce buffer before/after I/O completes. It does not have any restrictions on the original system buffer address.
The AMD Vi is the new IOMMU for AMD devices and also does not have any limits on the original buffer address. However on RHEL6.x kernels, the AMD IOMMU is disabled by default and must be enabled.
The following command line entries can be used to resolve the IOMMU issue:
'iommu=soft swiotlb=force' : Enable software IOMMU
'amd_iommu_enable=on' : Enable AMD Vi IOMMU
'mem=999G' : Limit amount of memory available to the kernel
Resources:
IOMMU Wikipedia entry
http://en.wikipedia.org/wiki/IOMMU