1. If the new gem is less valuable than any of your current gems, it is automatically sold and you receive gold instead.
2. If the new gem is more valuable than any of your current gems, the lesser gem is sold, and the new one takes its place.
You can also increase the inventory capacity by purchasing more gem slots.