A drysuit certainly makes me a lot more comfortable on the river. I have a pretty tight budget so I scrubbed around gearswap last winter until I found a used one at a price I could afford.
I have some nice fleece I use as a baselayer under the suit for really cold days.
Pogies are great for keeping your hands warm.
Get a neoprene beanie / helmet liner to help prevent the ice cream headache if you flip.