If the logo use was allowed in the contest and he gave permissions for this, then he also has no rights to ask you to remove this as well. He would have to have given very specific permissions for the logo use if he wants you to remove it. He'd have had clauses like "The logo is provided for use by the contestants, but is subject to removal once the competition closes, unless specifically stated otherwise.". If he simply said "use our logo, it's located here" and nothing else, then he gave away his rights.

Also you should let him know that he is getting free advertising, especially if you are getting search hits for his keywords.

But the main piece of advice for you is I recommend you hire a patents/trademarks attorney.