No, skulls are permanent, unless you collect them or move out of your dungeon.