Using CAT 6 for speakers in a patch panel

[sdl4000@xxxxxxxxx (simon)]

I have wired my newly built house with cat 6 cable; several in each
main room for AV and data. However, I was reading on one of these
forums some months ago that cat5 or 6 could be used, fairly
succesfully, as speaker wire - quality isn't too much of an issue, as
long as it sounds ok. So, I decided to also mount Cat6 sockets on the
walls ready to install speakers to. I've joined my cat cabling all to
a central patch panel in my garage and would like to be able to have
any device playing on speakers in any room.

My Dilemma

Now I'm trying to send a basic sound signal down them and I get no
joy, (I've played about swapping wires). Am I doing something wrong,
is there a wiring technique you should use for getting sound over
cat5/6? Can sound run from one device to another via a patch panel? Is
it just that I'm not kicking out enough power at one end?

Can anyone help?