The randomized stack frames technique is a technique that modifies the stack frames in an effort to thwart "return to libc" and similar attacks. Basically it hides valid addresses to the code from the attackers by either placing variably-sized padding between the address and the rest of the frame. or by storing these addresses encrypted It may not work on all platforms.
If the stack frame format is known, an attacker can extract the return address from the stack frame.
| - |
Encrypted return addresses
An instruction is placed at the entrypoint of the procedure, which XORs a random key into the return address thus encrypting it. The same instruction is placed before each exit from the procedure. Each procedure can use its own key to encrypt addresses in its frame. The keys |
| - |
Encrypted backlinks
Backlink is the pointer in the frame that points to the previous frame. There is a place (usually a CPU register) which holds the pointer to the current frame. When the backlinks in the frames are encrypted at the beginning of the procedure (by XORing them with a random key) and decrypted before each use, an attacker cannot walk through the chain of frames to analyze them. In the C and the P programming languages the backlinks in the frames are used only when the procedure is exitted from, so encrypting them does not have very significant impact on performance, especially when procedure calls are not executed very often. By contrast encrypting the current frame pointer is not a very good idea since it is used to access the local variables of the procedure. Should be used together with encrypted return addresses, otherwise the attacker could get the return address from the current frame. The keys used here can be different for different procedures and different from keys used to encrypt the return addresses. |
| - |
Current frame pointer displacing
The compiler randomly chooses a displacement value for each procedure to add to the current frame pointer at its beginning. The offsets of the local variables used by the procedure are adjusted so the procedure can use the modified backup pointer directly. The attacker does not know this displacement value so the value in the current frame pointer is almost useless for him. The backlink pointers also store these displaced current frame pointers but for added security they should be encrypted. |