I imagine that I experienced something very similar to your situation.
And it hasn’t been an issue for me since I did the exercise in the forum post I’m sharing.
If you do exactly as described in the post below, this won’t be an issue for you, not for a good while at least. 
The Big List: Location Ideas & Inspiration for Memory Palace Places
Good luck! And experiment! Play!
And enjoy yourself!
Regards,
Beau