I don't known the distribution you mention but... for some of the things
you talk of (like using samples as oscillators) did you have a look at
Pure Data? You can easily connect midi and audio do very creative/exotic
stuff. I'm sure if you're a 'hardware guy' (if I understand correctly)
patching should be fun.