No, essentially any well known structure real or imagined can be pressed into service.
The advantages of using a real building or location that is familiar is that you already have it memorized and perhaps having walked through it, Have a keen sense of the layout. It’s an easy way to get started.
Constructing your own fantasy palaces is more work but it offers more possibilities including the option of tailoring the structure to suit the information it is to hold. Also, in your imagination, size and space are not constraints. You can fit the Palace of Versaille into one compartment of a jewelry box and still traverse its chambers.
This thread has a lot of ideas about memory palaces:
https://forum.artofmemory.com/t/miniature-memory-palaces-including-your-palm/28864