I have not tried it myself specifically in rocksmith, but if you have Nvidia Shadowplay enabled graphics card, you can use GeForce Experience Shadowplay feature to record gameplay from any game that uses DirectX in fullscreen 1080p with as little as 5% loss of performance.