The land changed hands repeatedly. An empire governing a territory is not the same thing as the modern population living there.