Question is - what was the reason for refusing to answer a design-related question from a potential client?

If it was a public refusal, then it may leave a negative impression.
If it was a private refusal, then there is no reason why you cannot find clients again. It comes down to your level of skill and portfolio.