Born a slave in Louisiana, Cailloux was freed in 1846 and enjoyed success as a cigar-maker in New Orleans. He volunteered to join a regiment of Union soldiers forming in occupied Louisiana and died while attacking Port Hudson. He was the first black soldier to be killed in the Civil War.