The term actually means “unique.” It has no reference to number in this instance.
In Semitic language there is a complexity to words that does not occur in American English. Words in Hebrew, for example, are “terse.” This means that they can say complex things with layers of meanings in far less words than what is possible in English.
Because of the “terseness” of words in their language, the Hebrews began to develop a further level of complexities on top of these which were so loaded with meaning. This complexity was a series of plays on words. The most famous of these is the word “son,” which in Semitic conversation can mean more than one’s male child. It can often mean the embodiment of something else, its very essence, to be of “the same cloth.” In fact the expression “spitting image” comes from the Semitic use of “son,” especially in reference to how it used for and by Jesus Christ in reference to both God and Adam (as in “Son of God” and “Son of Man”).
“Only” is another one of these words. It doesn’t just refer to quantity. It also refers to quality. In this instance with Isaac it refers to how Abraham viewed him, how much he loved him, and how important he was in reference to God’s promise.
This is also the same meaning for the term about Jesus Christ when he is referred to as the “Only Son of God.” There are many “sons” of God. The angels are called “sons” of God. Human beings are referred to as “sons of God.” Adam is called “son of God.” But Jesus is the “Only Son of God” in that he is unique among “sons.” Only Jesus is God’ Son according to the terminology discussed above, i.e., the embodiment of God, God’s very essence, of the “same cloth” of God, God’s “spitting image.” Jesus isn’t the Son in the sense as being someone who is created by God and has a male gender, but is Son by essence of being the Second Person in the Godhead. This type of Son makes him unique, the “Only” Son of God.
It is this same sense for Isaac in that, though a literal son of Abraham, he is the most unique and important. Salvation history is tied up with this son, but not with any other of Abraham’s.