Why does invoking your right to remain silent, make you look suspicious?

I asked a question the other day, about innocent suspects remaining quite during an interrogation. Many feel an innocent person should be more then willing to explain their innocence. I can agree and disagree with that. Many people have a negative look at the police and are very caution when it come to this. Many times a persons story will be twisted and turned, and now the person really looks guilty. So many might err in caution. It makes sense to me. On the other hand police and others feel if you have nothing to hide, the just tell us what you were doing on such and such day, if you say nothing you must be hiding something. It is like a double edged sword. If we have this right, and we use it why can or does it make a person look guilty. It is not our job to prove our innocence, it is the LE, job to prove our guilt. I am just interested in your thoughts.