Only gems that are being merged during ritual could be stolen. Gems in the inventory or those that are simply installed into the totem (without being merged) cannot be stolen by other players.