Am i missing something or are you not able to turn down the difficulty just the speed?

Some don't have dynamic difficulty included because the charters prefer it that way.  You can add it yourself with the toolkit if you're so inclined.