Only gems that are being merged during the ritual can be stolen. Gems that are in your inventory or that are simply placed in the totem (without being merged) cannot be stolen by other players.