I’m reading a fantastic book right now called The Civil War as Theological Crisis, by Mark A. Noll. It spends two chapters analyzing this very issue, with a third chapter dedicated to Catholic perspectives.
In my opinion, yes: the Bible (and Catholicism) suggest that there are circumstances under which it is morally permissible to strip a person of liberty and subject them to strenuous labor without pay - that is, to enslave him. The most common example of this today is imprisonment, where convicted criminals are stripped of freedom and forced to do a variety of useful tasks while they pay their debts to society. POW’s often face similar circumstances. In history, another common form of this was indentured servitude, where one party willingly sold his freedom temporarily in exchange for later reimbursement.
What the Catholic Church has unequivocally condemned throughout history (with the support of the Bible, even if American Protestants north and south were too blind to realize it) is the race-based chattel slavery of the sort practiced in antebellum America. The enslavement of complete innocents has always been understood to be wrong.
Short answer: yeah, the Bible supports slavery. So does your brother, if he believes in the modern prison system. But that thing he means when he says, “slavery”? That really bad thing we had in the South before the Civil War? That is incompatible with a fair reading of the full Bible, and is certainly and clearly condemned by the official, centuries-old teachings of the Church.