The design certainly isn't the problem in my opinion.
Promoting a site like this can be difficult as you need to find people with nothing better to do than enter your contests. This means aiming for people who are bored/young/crazy enough to be interested. Try advertising on sites that have a similiar target audiance to you, humor sites etc. If you know of any large general forums for that kind of thing stir up a frenzy about one of your contests.
It's not going to be easy due to the randomness but once you've managed to whirl up a storm (and people know you are legit) ad you have past contest entries to showcase on your site then it will boom I think.
Good luck with it, let us know how you progress