Just over one month after the first shots of the Civil War were fired, three enslaved black men got into a row boat and paddled across the James River from mainland Virginia to the Union-occupied Fortress Monroe. Whether they knew it or not, the three young men – named Frank Baker, Shepard Mallory, and James Townsend – sparked the unraveling of the institution of slavery in the United States. In today’s installment of our ‘do-over’ series, we’re revisiting the complicated legal category of contraband, the term applied to enslaved people who fled to Union lines during the American Civil War.
Sarah: On May 23, 1861 – just over one month after the first shots of the Civil War were fired at Fort Sumter, South Carolina – three enslaved black men got into a row boat and paddled across the James River from mainland Virginia to the Union-occupied Fortress Monroe. They had been taken from their homes and pressed into the service of the Confederate Army, forced to build artillery fortifications across the river from the Union lines. While they worked, building artillery placements for an army organized and deployed for the sole reason of protecting the institution of black chattel slavery, over their heads would have fluttered the banner of the 115th Virginia Militia, emblazoned with the words: “Give me liberty or give me death.”
Marissa: If that irony wasn’t enough for you, consider that the spit of land where they labored was also the same place where the first enslaved Africans arrived in the colonies all the way back in 1619. With the courage of these three men, America’s peculiar institution began to die in the same place where it was born.
Sarah: Whether they knew it or not, the three young men – named Frank Baker, Shepard Mallory, and James Townsend – had sparked the unraveling of the institution of slavery in the United States. If I had to guess, I would say that most of you listening likely already have a narrative about the end of slavery in your mind. It might start sometime around 1859 with John Brown’s raid on Harper’s Ferry, or maybe in 1861 with the first shots of the Civil War, but I would venture that it probably hinges on one moment in political history: Abraham Lincoln and the Emancipation Proclamation. After all, Lincoln freed the slaves, right? He’s the “Great Emancipator!”
Marissa: But over the past ten or twenty years, historians have been dismantling this foundational myth of American history. Don’t get us wrong – Lincoln played a pivotal role in ending slavery. But his role was more cautious, more reluctant, and less charitable than we often believe. Instead, many historians argue, it was enslaved people themselves who voted with their feet, taking it upon themselves to escape enslavement and demand freedom at the Union front lines.
Sarah: But freedom didn’t come easily. Not only did enslaved people like Frank Baker, Shepard Mallory, and James Townsend risk their lives when they escaped from the Confederate Army or their enslavers to seek a new life on the Union lines, they were not always met with open arms. Instead, they might face anything from impressment into the Union Army, to a purgatory-like existence in vast, impromptu camps with poor sanitation, short supplies, and racist military guards. In today’s installment of our ‘do-over’ series, we’re revisiting the complicated legal category of contraband, the term applied to enslaved people who fled to Union lines during the American Civil War.
I’m Sarah
And I’m Marissa
And we are your historians for this episode of DIG.
Promos?
Sarah: When the Civil War began in 1861, slavery was the problem at its very heart. (If that statement is surprising or upsetting to you, you need to go do some more reading – and I’m more than happy to give you reading recommendations – but I’m not going to build that case here and now.) But while slavery was the driving cause of the war, that does not mean that every actor in the Civil War had the same or similar feelings on the issue of slavery. One of the things that some people believe about the Civil War is that Abraham Lincoln hated slavery and was elected to put it to an end – but that’s really not accurate at all. When Abraham Lincoln was inaugurated in March of 1861, he made it very clear in his first inaugural address that he was not going to be an abolitionist president. Within the first few lines, Lincoln declared that he had “no purpose, directly or indirectly, to interfere with the institution of slavery in the States where it exists.” Lincoln explained that he had been elected to uphold the Constitution, and since slavery was protected by the Constitution and executive power did not give him the power to change that, it was his job to govern the nation as it was. He summed it up this way: “One section of our country believes slavery is right and ought to be extended, while the other believes it is wrong and ought not to be extended. This is the only substantial dispute. The fugitive- slave clause of the Constitution and the law for the suppression of the foreign slave trade are each as well enforced, perhaps, as any law can ever be in a community where the moral sense of the people imperfectly supports the law itself. The great body of the people abide by the dry legal obligation in both cases, and a few break over in each. This, I think, cannot be perfectly cured, and it would be worse in both cases after the separation of the sections than before. The foreign slave trade, now imperfectly suppressed, would be ultimately revived without restriction in one section, while fugitive slaves, now only partially surrendered, would not be surrendered at all by the other.”
Marissa: What’s notable about the first inaugural address is that he’s speaking not to the nation as a whole, like in most inaugural speeches, but directly to the South, particularly because by that March, several states (SC, MS, FL, AL, GA, TX) have already seceded. And what Lincoln is trying to do in this speech is convince the South that he’s not going to abolish slavery. He’s saying that the real argument is over the extension of slavery into the territories, not over whether to abolish slavery as it existed in the south. Historian Allen Guelzo, among others, have pointed out that Lincoln was being very lawyerly – he never said he would abolish slavery, but he also didn’t defend the morality of slavery and parsed his words in some places about slavery and the Constitution in ways that pissed off both Southerners and Northern abolitionists. And we know that Lincoln will eventually take major steps to end slavery in the United States. But the key takeaway here in March 1861 is that Lincoln did not blaze into office chomping at the bit to end slavery. The other thing that Lincoln made clear in the first inaugural was that if there was going to be a war, Southerners were going to have to be the aggressors. Just before he ends the speech, he says this: “In your hands, my dissatisfied fellow-countrymen, and not in mine, is the momentous issue of civil war. The Government will not assail you. You can have no conflict without being yourselves the aggressors. You have no oath registered in heaven to destroy the Government, while I shall have the most solemn one to “preserve, protect, and defend it.” This phrase really shows Lincoln’s ability to construct persuasive and clever arguments. By saying that the South must be the aggressors, he understood that it meant that the South would have to take the blame for starting a war. At the same time, though, Lincoln understood that it meant that that North could fight a war that could look as though it was completely motivated by the desire simply to reunite the broken Union – not by a zealous desire to destroy the institution of slavery.
Sarah: But while Lincoln was careful not to make the war (or, at that point, potential war) about slavery, others were determined to make sure that slavery took center stage. An example of this that I use with my students is two Union marching songs. There were tons of marching songs, and many of them were just fun or funny songs to sing, but two in particular stand out to me as examples of how no matter how much Lincoln tried to control the image of the Union’s motivations during the war, slavery remained at its center. (Side note: it’s funny that both songs use the same tune!) The first song is John Brown’s Body, which talks about the soul of John Brown “marching on,” just as the Union army marched onward into battle. The final stanza (there are multiple versions) goes:
He captured Harper’s Ferry with his nineteen men so true
He frightened old Virginia till she trembled through and through
They hung him for a traitor, they themselves the traitor crew
But his soul goes marching on
Glory, Glory, Hallelujah
Glory, Glory, Hallelujah
Glory, Glory, Hallelujah
His soul goes marching on
The second song is probably the most famous – and is still often sung in churches or at patriotic gatherings: The Battle Hymn of the Republic. This song, written by Julia Ward Howe, is about how the Union army marches forward to enact God’s truth – not just any truth, but the truth that slavery must end. And the song ends with one of, I think, the most moving bits of poetry written about this war:
He has sounded forth the trumpet that shall never call retreat;
He is sifting out the hearts of men before His judgment seat;
Oh, be swift, my soul, to answer Him; be jubilant, my feet!
Our God is marching on.
In the beauty of the lilies Christ was born across the sea,
With a glory in His bosom that transfigures you and me;
As He died to make men holy, let us die to make men free!
While God is marching on.
[Conversation? I find these two really powerful, to the point that sometimes when I read them out loud with my students I actually get a tiny bit choked up.]
Marissa: All of this – the inaugural address and the songs – is just to show that while we know that slavery was the core cause and problem of the war, not everyone at the time agreed that that was what the war was about as it was happening. While Lincoln, especially early in the war, wanted to play his hand close to the vest in terms of his plans for slavery, instead arguing that the sole purpose for the Union’s fight was to preserve the Union, that doesn’t mean that others agreed with that assessment. And those who did the most to ensure that the war was centered on the issue of slavery and freedom weren’t the Union soldiers singing John Brown while on the march or the politicians parsing words in the capitals, but enslaved people themselves. From the moment the first men in Union blue appeared in slaveholding states, enslaved people knew that this war represented their opportunity to seize freedom. When Frank Baker, Shepard Mallory, and James Townsend rowed across the James River to Fortress Monroe, they were forcing the Union Army, and by extension, Abraham Lincoln, to address slavery directly. It was enslaved people themselves more than any marching song that made the war a war to destroy the institution of slavery.
Sarah: So let’s come back to that moment when they arrived at the fort on the coast of Virginia. The next morning after they arrive, they were brought before the commander of the fort, Benjamin Butler. Butler was an odd duck. He had absolutely no military background, but saw service in the military as a way of potentially advancing a political career so lobbied hard for a commission in the Union Army. John William DeForest, an officer who served under Butler later in the war, recalled that “on the whole, he seemed less like a major general than like a politician who was coaxing for votes.” Butler liked to stir things up and because of that, he presides over some of the more dramatic moments of the war and several more after the war. (He was the guy who probably slept with Victoria Woodhull!) Another critical thing to know about Ben Butler is that he was not an abolitionist – he was a Democrat, a party that owed much of its success to the white working classes that thought that abolitionists were dangerous zealots bent on race amalgamation. So when these three men came before him, he wasn’t naturally inclined to be in their favor. But at the same time, he was presiding at the fort over a large number of new Union recruits, many from Massachusetts, who were abolitionists or the sons of abolitionists. If Butler dismissed the young enslaved men back to their captors, it was likely his soldiers wouldn’t be very happy with their commander. Like the ambitious politician he was, Butler “read the room,” so to speak.
Marissa: Butler asked the young men a bunch of questions trying to figure out the best path forward. Where were they from? Who were their masters? What work had they been doing? Ultimately, he sent them away without any indication of his position. Just a few minutes later, an adjutant came to the general to say that a Virginian, under a flag of truce, had come to the Fort to speak to him. The man, it turned out, was Major John Baytop Cary of 115th VA – the unit that had been holding the three enslaved men. Butler went out and rode out to meet Cary, and they had a conversation on horseback. I’m just going to read a description of the conversation they had here: (This conversation was reconstructed by historian Adam Goodheart, and just so you know, it includes language that we wouldn’t use today)
““I am informed,” he said, “that three Negroes belonging to Colonel Mallory have escaped within your lines. I am Colonel Mallory’s agent and have charge of his property. What do you mean to do with those Negroes?”
“I intend to hold them,” Butler said.
“Do you mean, then, to set aside your constitutional obligation to return them?”
Even the dour Butler must have found it hard to suppress a smile. This was, of course, a question he had expected. And he had prepared what he thought was a fairly clever answer.
“I mean to take Virginia at her word,” he said. “I am under no constitutional obligations to a foreign country, which Virginia now claims to be.”
“But you say we cannot secede,” Cary retorted, “and so you cannot consistently detain the Negroes.”
“But you say you have seceded,” Butler said, “so you cannot consistently claim them. I shall hold these Negroes as contraband of war, since they are engaged in the construction of your battery and are claimed as your property.”
Sarah: We’ll explain all that banter about seceding or not seceding soon. The next day, Butler sent a note to Washington to update the War Department of the decision he had come to regarding the three men. But even before Lincoln and his cabinet could respond, Butler faced the immediate consequences of his decision. He had talked to Cary on Friday. By Sunday morning, eight more enslaved men appeared at the fort’s gates. On Monday, that eight was followed by 47 more – and this time, it wasn’t just young men, but families, elderly people, and mothers with small children. According to historian Adam Goodheart, by Wednesday, one Massachusetts soldier wrote home that “slaves are brought in here hourly.”
Later, in July 1861, he would elaborate on his decision in a letter to Secretary of War Simon Cameron. Butler explained that initially, it seemed very clear that men working in the name of the Confederate Army could easily be considered contraband, since they were being “used in aid of rebellion,” just as a ship or cannon might be. But it became more complicated when he realized that women, children, and elderly people were also flocking to the perceived safety of the fort. They couldn’t be considered contraband in the same sense, and if they were ‘confiscated’ as ‘property,’ it would mean that the Union Army believed they really were propert – unless the act of escaping and running to Union lines, where slavery was not recognized, they set themselves free. Butler asked Cameron: “Have they not become, thereupon, men, women, and children? No longer under ownership of any kind, the fearful relics of fugitive masters, have they not by their masters’ acts and the state of war, assumed the condition which we hold to be the normal one, of those made in God’s image? It not every constitutional, legal, and moral requirement, as well to the runaway master as their relinquished slaves, thus answered? I confess that my own mind is compelling by this reasoning to look upon them as men and women. If not free born, yet free, manumitted, sent forth from the hand that held them never to be reclaimed.” Believing that this was the case, Butler explained that it was his duty, as a “humane man,” to sheltered even women and children who did not directly support the Confederate cause.
Marissa: But Butler was concerned that his reasoning wouldn’t be enough because not long before, Brigadier General Irvin McDowell, the general in charge of the Army of Northeastern Virginia, had issued an order declaring that runaways who came to the Union lines should be turned away. Did this just apply to the Army of Northeastern Virginia? Did it mean that slaves would be treated as fugitive slaves – which, constitutionally, needed to be returned to their masters? How were military commanders supposed to determine a refugee’s status? If the Secretary of War demanded that McDowell’s order be followed, Butler said, he would comply. But if he was given the ability to act independently as an officer, he would follow his own reasoning. He wrote: “In a loyal state, I would put down a servile insurrection. In a state of rebellion, I would confiscate that which was used to oppose my arms—and take all that property which constituted the wealth of that state, and furnished the means by which the war is prosecuted, besides being the cause of the war; and if, in so doing, it should be objected that human beings were brought to the free enjoyment of life, liberty, and the pursuit of happiness, such objection might not require much consideration.”
Sarah: Whether Butler realized it or not, he had forced Secretary of War Cameron and the entire Lincoln administration is a tight spot. Cameron wrote back on August 8 that he believed eventually Congress would settle question and that in the meantime, Butler could continue to accept runaways – but he was also firmly instructed to report back to the Washington often, to ensure that his soldiers did not encourage enslaved people to leave “the lawful service of their masters,” and that he would never prevent the “voluntary return of any fugitive to the service from which he may have escaped.” As Marissa mentioned a moment ago, and as Cameron was getting at the end of his letter, Butler had created a constitutional problem. This gets a little complicated to explain, so bear with us. The Constitution includes a clause in Article 4 Section 2 known as the fugitive slave clause. The clause reads like this: “No Person held to Service or Labour in one State, under the Laws thereof, escaping into another, shall, in Consequence of any Law or Regulation therein, be discharged from such Service or Labour, but shall be delivered up on Claim of the Party to whom such Service or Labour may be due.” That’s why, under the constitution, Ben Butler should have returned those enslaved men to their masters when they came calling. But Sarah, you might be saying, that doesn’t make any sense, because by the time the three young men arrived at Fortress Monroe, Virginia had already seceded and was adding itself to the new Confederate States of America! You’re not wrong – but you’re also not totally right. This is where it gets a little tricky, and where we actually need to take a step back to look again at Lincoln’s first inaugural address.
Marissa: In the inaugural address, Lincoln vows to uphold and enforce the Constitution, which includes the fugitive slave clause. As he states right from the beginning: “I have no purpose, directly or indirectly, to interfere with the institution of slavery in the States where it exists. I believe I have no lawful right to do so, and I have no inclination to do so.” But how does that work when several states have just … left the union? Well, Lincoln would say, they didn’t leave the union because you can’t leave the union. For a major chunk of the address, Lincoln actually turns into a little bit of a Constitutional Law professor to explain how secession is both illegal and impossible. This comes down to a fundamental disagreement between Lincoln and most Southerners on what the United States actually is. Many Southern politicians believed that the United States was actually a confederation of states – that the states were all sort of independent actors that all came together to establish a social compact to have a federal government. And when that federal government overstepped, they believed that states had a kind of sovereignty to just … opt out. This is how South Carolina almost seceded in 1833 during the Nullification Crisis – they just said they didn’t like a federal law, said they had the power to nullify it, and threatened to take their marbles and go home if the federal government didn’t let them. This was summarized by South Carolina senator John C. Calhoun, who argued that the US “government if federal, in contradistinction to a national, government,” referring to the concept of federalism, or the sharing of power between state & federal governments.
Sarah: But Lincoln argued that Calhoun and his Southern brethren had it all wrong. The states didn’t create the union, the union created the states. After all, he argued, the “Union is much older than the Constitution,” and was formed all the way back in 1774, solidified in the Declaration of Independence in 1776, and further shaped by the first governing document, the Articles of Confederation. After all, he says, “in 1787, one of the declared objects for ordaining and establishing the Constitution was to “form a more perfect Union.” In other words, the union existed long before states did, and therefore states don’t have the right or the ability to revert to a previous state of independence – that previous form would be as a colony, not as a state. The argument that Lincoln is building here is that secession is not a thing. You can’t secede. You can say that you secede, but that Constitution doesn’t include an escape clause, so actually, you remain as you always were in terms of your relationship with the federal government. You’re not a foreign country. For instance, Lincoln says that the mail (a federally controlled government service outlined in the Constitution) will still be delivered! And he goes on to say that the legal recourse for being angry about government policy is to amend the Constitution, but that secession, if it were possible, would be the essence of anarchy.
Marissa: So while Butler seems to indicate in his letter to Cameron that he knows this was a tricky legal situation, it actually had much larger implications than he probably suspected. By refusing to return fleeing slaves, he was actually acting in conflict with the United States Constitution. If Virginia had never actually seceded, than the Fugitive Slave Clause was still good law – and Butler should have returned the enslaved men. By refusing to return those enslaved men, Butler was playing into Confederate talking points that Lincoln was a sneaky, aggressive zealot bent on abolishing slavery. Butler had also undermined the argument Lincoln made in the first inaugural that he was going to act as though the southern states had never seceded, and continue to enforce the law of the land. After all, Butler is an agent of the US government, effectively seizing private property without solid legal footing. Moreover, it wasn’t clear what this meant outside of Fortress Monroe. Was this the official stance of the United States government? What should other commanders do? As we see from Butler’s confusion over McDowell’s order to his army, different officials were acting independently and sometimes issuing clashing orders – things were sort of a mess.
Sarah: The issue was sort of addressed, as Cameron suggested, by Congress in the form of the Confiscation Act of 1861. We’re going to refer to this as the First Confiscation Act, because there were two laws with the same name, and we’ll come to the second one soon. This law was signed into law by Lincoln on August 6, 1861, so it was in its first days when Cameron wrote back to Butler, and it would have addressed at least some of Butler’s concerns. The law allowed for the confiscation of any property being used to support the Confederate cause as “contraband of war.” This was on solid legal ground, with long precedents in European war traditions and solidified in the fairly recent 1856 Declaration of Paris. But from the beginning, the intention of the law – as influenced by Radical Republican Senator Lyman Trumbull – was to include enslaved people as ‘contraband of war.’ This was pretty transparent, and even as it was debated in Congress, it sparked the same kind of accusations of abolitionism that Butler’s actions had – very accusations that Lincoln was trying to avoid. Lincoln was very reluctant to sign the bill, largely because he believed it would never pass constitutional scrutiny. There was shaky legal ground, at best, for arguing that enslaved persons were contraband of war, especially women and children. It wasn’t that Lincoln didn’t believe that there was, as Allen Guelzo phrases it, a “humanitarian appeal” to the Confiscation Act – but he believed any long-term effort toward legally sound emancipation would be damaged by such bold and ill-founded projects. This is where Lincoln can be such a tricky, complicated historical figure – he wasn’t proslavery, or even ambivalent to slavery, but he did think that any effort to end slavery had to be done slowly, deliberately, and with very strong legal grounding, lest it all crumble and allow slavery to be even more entrenched than before. But that meant that when enslaved people were arriving at the very gates of freedom by ‘voting with their feet’ and going to Union lines, Lincoln wasn’t tripping over himself to accommodate them … which isn’t great. However reluctantly, Lincoln did sign the bill, and it became law in the late summer of 1861.
Marissa: Things actually got even more confusing shortly after Cameron’s early August letter back to Butler. In early July, Lincoln had commissioned the famous western adventurer and sometime-politician, John C. Frémont, to head up the Department of the West, the military district the covered everything from the Mississippi River to the Rocky Mountains. Primarily, Frémont’s real job was making sure that slave-holding Missouri did not join the Confederacy, so he set up his headquarters in St. Louis. By the time he arrived, though, Missouri was in utter chaos as the state devolved into pro-Union and pro-Confederate factions. (We’ve covered some of this story before, so take a listen to our Civil War Guerrillas episode.) Frémont thought the best way to establish his control was by declaring martial law in St. Louis when he arrived in mid-August 1861, over the complaints of the Unionist government officials already in the city. Just over a week later, Frémont took things a step further by declaring martial law across the entire state of Missouri. What’s more, he issued an order, influenced by the very newly signed Confiscation Act, that said that “the property, real and personal, of all persons … who shall take up arms against the United States, or who shall be directly proven to have taken an active part with their enemies in the field, is declared to be confiscated to the public use, and their slaves, if any they have, are hereby declared free men.”
Sarah: For most of that order, Frémont was more or less on solid ground. After all, the Confiscation Act allowed the United States government to seize ‘contraband’ property, including enslaved people. He was pushing things a bit – he was declaring this power through the imposition of martial law, which was concerning and a bit tyrannical, and expanded the criteria by which property could be considered ‘contraband.’ (In other words, not just property used in the continuation of the cause, but person property, and even the personal property of those who even had a tenuous connection to the cause *Guelzo’s example of a Confederate soldier’s wife’s piano). It was the last phrase that really took things to a new level. Butler’s Fortress Monroe decision and the Confiscation Act both allowed for the seizure of contraband goods, but never touched the question of whether enslaved people considered contraband were free. Emancipating enslaved people was taking the legal tap-dance about war powers over enemy property to a completely different place. Did the federal government have the power to just … free slaves?
Marissa: This was another problem for Lincoln. Again, it’s not that Lincoln loved the idea of slavery or that he wanted to protect it. But during the chaotic, confusing first months of the war, Lincoln’s first priority was not freedom – it was actually keeping the border states from seceding and joining the Confederacy(Kentucky, Missouri, Maryland, Delaware; eventually also Kansas, which became a state in 1861, and West Virginia, which became a state in 1862). Each of these states was slaveholding. Lincoln and his administration were extremely sensitive to the fact that any effort on their part to end slavery to any degree would be interpreted by the border states as an act of aggression, and very likely drive them into the arms of the slavery-obsessed Confederacy. Frémont’s very job in the Department of the West had been to keep Missouri from joining the Confederacy – but here he was making a decision that put that in jeopardy. And it wasn’t just Missouri that was affected by Frémont’s act. Joshua Speed, Lincoln’s former law partner and friend, wrote to the president to explain how the act was going to be interpreted in Kentucky, his home state. “I have just seen Frémont’s proclamation. It will hurt us in Kentucky. The war should be waged upon high points and no state law be interfered with – our Constitution & our laws both prohibit the emancipation of slaves among us – even in small numbers – If a military commander can turn them loose by the thousand by a mere proclamation.- it will be a most difficult matter to get our people to submit to it.” Speed also pointed to another fear those in border states would have – that this order would lead to huge numbers of enslaved people fleeing their homes and acting as if they were free, which raised the specter of slave rebellion to many paranoid Southerners. Speed wrote: “All of us who live in the slave states whether Union or loyal have great fear of insurrection. Will not such a proclamation read by the slaves incline them to assert their freedom?”
Sarah: Historian James McPherson has described Lincoln as an iron fist in a velvet glove, and his handling of Frémont is a good example of that. Ever the diplomat, Lincoln wrote to Frémont in early September asking him not to rescind the order, but modify, or at least not to enforce, its more radical aspects. Frémont wrote back more or less telling Lincoln he would do no such thing. This rubbed Lincoln the wrong way, and he wrote back and this time more pointedly directed Frémont to modify the order. In addition, he sent two allies, cabinet member Montgomery Blair and quartermaster general Montgomery Meigs (what is the likelihood of two Montgomerys?!) to Missouri to report back on Frémont’s conduct. Their report was not great – in fact, Blair reported that Frémont “seems stupefied & almost unconscious & is doing absolutely nothing.” Frémont saw where this was going, and tried to use his political connections to keep him in his post, but by late October dismissed Frémont from his post. Antislavery politicians and activists were not happy about Frémont’s dismissal. While Frémont was by no means an abolitionist hero, his action – like Butler’s – seemed like the kind of bold, decisive blow against the slave power that the nation needed. But Lincoln was determined to move carefully and deliberately on the issue, even if it meant angering members of his own party.
Marissa: Lincoln’s position here, from a modern perspective, doesn’t look great. There’s absolutely an argument for saying that immediate, bold emancipation, in whatever ways possible, would have been better than biding one’s time for the right moment. But just to offer a little perspective on Lincoln’s position, we want to quote directly this summary from Allen Guelzo – and you by no means need to agree with it. Guelzo explains that all these efforts were contingent on a state of war, had no permanence, and did nothing to affect the legality of slavery as an institution. “Which is why Lincoln scarcely ever mentioned contrabands, was skeptical about the Confiscation Act, and “modified” Frémont’s proclamation: None of them had any promise that they would stick. And if any one of them failed to survive a curt challenge, the prospect for all future emancipation would be set back just as Taney’s decision in Dred Scott has set back the struggle to keep slavery out of the territories. And this was before any consideration of the possible political debris these schemes would shake out in the border states or in Congress. In the wake of Bull Run recalled James Blaine, “the military situation was so discouraging that in the president’s view it would have been wiser for Congress to refrain from enacting laws which, without success in the field, would be rendered unnecessary.”
Sarah: And Lincoln had his own plans for ending slavery while keeping the border states in the Union – such as a compensation scheme that would reimburse slaveowners for their lost property. That would shift and change over the course of the war, and as we all know, or should know, Lincoln eventually moved in September 1862 (a year after all these debacles) to issue the Preliminary Emancipation Proclamation, which used the ‘success’ (to be generous) of Antietam to sell the idea of emancipating slavery in the afterglow of a military victory. But we’re running low on time, and it would be a dereliction of our mission as Dig to use this whole episode to discuss the political machinations of several powerful, white men, so I want to shift our perspective a bit. At the heart of the issue of contraband wasn’t legal definitions or political decision making, but enslaved people making difficult, often life-threatening, even life-ending, decisions. After all, the people who forced slavery to the fore in the first year of the war weren’t really Ben Butler and John C. Frémont, but the three men who escaped from bondage in Virginia and sought freedom at Fortress Monroe.
Marissa: As we already mentioned, within days of those men’s arrival at the fort, dozens more enslaved people arrived seeking refuge. By the end of 1865, Fortress Monroe is home to what becomes known as the Grand Contraband Camp, home to some 10,000 refugees of slavery. Camps like the Grand Contraband Camp pop up everywhere that Union Army goes – refugees tag along even when the army is on the march. According to historian Amy Murrell Taylor, refugees established nearly 300 camps across the south, from coastal Virginia to Kansas. But just because refugees gathered near Union encampments, forts, and lines does not mean that they were safe or comfortable. Camps often were overcrowded, suffered from poor sanitation, shelter was sometimes scare, and supplies were always low. Enslaved people often fled bondage with with nothing but the clothes they had on their backs – which typically wasn’t much. John Eaton, a union officer who oversaw the Union response to the refugees in the Mississippi River valley, wrote in 1864 that “you saw them, of both sexes, of all ages, in every stage of health, disease, and decrepitude, often nearly naked, their flesh torn in escaping.” A woman named Maria Mann, who worked as a teacher of freedpeople, wrote in 1863 that refugees “arrived in Union camp with swellings, open sores, and eaten up with vermin … their mortality is greatly increased.” Decades after emancipation, one former refugee recounted her experience with the flight to freedom: “We was freed and went to a place that was full of people. We had to stay in a church with about twenty other people and two of the babies died there on account of exposure. Two of my aunts died, too, on account of exposure then.”
Sarah: Amy Murrell Taylor, in her fantastic and multi-award winning new book, Embattled Freedom, tells the story of Emma and Edward Whitehurst, who escaped enslavement in Virginia to Ben Butler’s federal troops in Northern Virginia. They gathered on a plantation that Butler’s troops had seized near Newport News. When they arrived, they were inspected to determine to what extent they could perform manual labor. As much as Union Army and federal government officials were concerned that refugees would be little more than a drag on the army’s resources, they were more than happy to benefit from their manual labor (while, of course, denying that labor to their enemies). Edward was pressed into labor as a hospital steward and gravedigger, while Emma likely did laundry or cooked for the troops and other refugees. But within a few weeks they experienced another danger of seeking refuge with the Union Army: the constant threat of re-enslavement by the encroaching Confederate Army. In late July, Irvin McDowell and his Union force was beaten badly by Confederate forces led by PGT Beauregard and Joe Johnston. While that battle was about a hundred miles away from their location in Newport News, the defeat threw the Union Army into disarray and broke up Butler’s forces on the coast leaving the refugees vulnerable. In an effort to capitalize on the Union Army’s quick evacuation of the coast, and to recoup the lost property in slaves under Union protection in the area, Confederate General John B Magruder set out to march up the coast. Magruder gave those in his command the following orders: “If the negroes in the Back River region and on the James River can be surprised and captured at night or by day by small parties of troops, let them do it.” Some were captured – at least 150 refugees were captured and sent back into enslavement. Emma and Edward Whitehurst were able flee to the safety of Fortress Monroe in time (along with thousands of other refugees) but the threat of Confederate capture was very real wherever the refugees stayed with the federal army.
Marissa: So-called “contraband” people were in a legal limbo in terms of their status. Other than Frémont’s short-lived order declaring contraband slaves free men, none of the other orders or acts (like the Confiscation Act) addressed their free or slave status. Refugees like Emma and Edward were not really slaves anymore, but weren’t entirely free, either. Their status was made even more murky (well, Edward’s was) by another law passed by Congress in July 1862. This law, called the Second Confiscation and Militia Act, broadened the first Confiscation Act by allowing the Union Army to take any and all personal property from ‘rebellious persons.’ Actually, the law as written stated that any person “committing the crime of treason” could be fined, imprisoned, or executed, and their slaves set free. Lincoln wasn’t nuts about this act either, but after some haggling, eventually signed it, too. But along with the Second Confiscation Act also came the Militia Act, which authorized the US military to use black men in the service of the military for manual labor, camp service and as soldiers. In exchange for this service, black men – here we’re mostly talking about contraband/refugees – and their immediate families would be rewarded with freedom.
Sarah: This seems like a real step in the right direction, right? This law eventually leads to the creation of the first black troops, like the famed 54th Massachusetts and later the creation of the United States Colored Troops. We talked about the black military experience during the Civil War in our episode on Black Union Soldiers, so we won’t go into that whole story here. And, even better it seems, in January 1863, Lincoln issued the full Emancipation Proclamation, which freed all enslaved persons in the states in active rebellion. And one of the dominant theories of black military service is that it was a transformative experience, making former slaves into potential citizens. As one famous quote from a black soldier goes: “I’s a man now.” And the Emancipation Proclamation, the story goes, brought about the Day of Jubilee, the longed-for, fought-for, bled-for day when slavery died. But while it seems like a step in the right direction, and indeed, maybe the end of the story of slavery and emancipation, the lived experience of refugees was still complicated and, often, tragic. (Of course, right?) So I want us to finish today with a story about one refugee family to give you an idea of what this all actually looks like.
Marissa: Outside of Lexington, KY, there is a Union fort called Fort Nelson. By 1864, 500 slaves had poured into the fort, establishing a contraband camp. Many of the refugees at Fort Nelson were people with freed status from Tennessee – by virtue of their bondage in a Confederate state after the Emancipation Proclamation, Tennessee’s slaves were technically free. But many others at Fort Nelson were from Kentucky, and because of they happened to be held in a border state, their status was ambiguous – sort of contraband, sort of slave, definitely not free. One refugee family with this ambiguous status that sought freedom at Fort Nelson November 1864 was the Miller family: Joseph Miller, his wife and four children. By the time they arrived at the fort, Miller’s seven year old son was ill, likely from exposure on the cold, wet trek to freedom. The Union Army officials processing incoming refugees allowed the family to enter the safety of the fort, but on one condition: Joseph Miller had to enlist in the Union Army – in exchange, they said, they would provide safety, shelter, and food to his family. Of course, Joseph Miller agreed to the bargain and became a US soldier.
Sarah: This complicates the story we’ve heard so far. No where in the Confiscation or Militia Act did it say that ‘contraband’ had to perform labor in exchange for protection, especially not service in the military. Miller was effectively pressed into service in exchange for his family’s care. And while it wasn’t uncommon for armies or governments to offer protection for a soldier’s family when he enlisted and went off to war – this was a major promise extended to Confederate soldiers, for instance – the circumstances were vastly different. Miller’s family was standing in the cold and rain outside a fort in a slaveholding territory. If the army didn’t take them in, they faced extreme danger. Nonetheless, this arrangement must have been a sigh of relief for the Miller family – yes, Joseph would have to join the army, which would be scary and dangerous, but it could also be a huge moment where he could assert himself as a man and not a slave, right? And the family would have someplace to go, food to eat, shelter from the November weather and maybe even military medical care. But it wasn’t long at all before the Union officials at Fort Nelson went back on their promise to Joseph Miller and the other black soldiers pressed into service. Brigadier General Speed S. Pry, commanding the fort, ordered his white soldiers forcibly evict all the refugees, except the men who had joined the army, from the fort. The soldiers began the evictions in the middle of the night in late November while the freedpeople were sleeping. Panic and confusion set in as many of the freedpeople mistook the Union soldiers breaking down tents and screaming orders for attacking Confederates. The white soldiers forced the refugees out, in the middle of a cold, rainy night, with only the clothes and supplies they could carry on their backs.
Marissa: Among the families evicted was the family of Joseph Miller, who had only had a handful of days of freedom at the fort. Miller was horrified that the army that he had just joined was demanding his family leave. His son was still sick, and surely sending him out into the elements would spell disaster. Miller later recalled of the experience: “I told the man in charge of the guard that it would be the death of my boy. I told him that my wife and children had no place to go. I told him that I was a soldier of the United States.” The guard had no sympathy – quite the opposite. He told Miller that if his family didn’t evacuate their tent and get on the wagon removing freedpeople from the fort, he would shoot “every last one of them.” Miller had no choice but to watch as his wife and four children, all between 4 and 10, get on a wagon headed out of the safety of the fort and back out into the danger of slaveholding Kentucky. Later, Joseph Miller was able to leave Fort Nelson and venture out in search of his family, who he finally found six miles away, packed into an old boarding house run by free people. His wife and three of their children were huddled a corner of a room, freezing and hungry. He soon learned that his seven year old son, the one who had already been sick, died on the trip to the boarding house, sitting in an open wagon in the freezing rain. But as a soldier, under orders from his officers, he couldn’t stay with his family – he had to return to Ft Nelson.
Sarah: The next morning, Miller again walked back to the boarding house, dug a grave for his son, and then returned again to Ft Nelson, unable to even ensure that his wife and remaining children would have access to food or even adequate heat. And consider that in about 48 hours, he’s walked something like 18 miles in cold, wet weather, learned of his son’s death, dug his grave, and buried him. What comes next shouldn’t be much of a surprise: within a couple of weeks, Miller’s wife Isabella and his son Joseph Jr. died. Ten days after that, his daughter Maria died. Finally, his son Calvin died. And just five days after that, Joseph Miller himself died, within the safe confines of Fort Nelson. I’m going to read directly from Jim Downs’ account of the Miller family here: “The Miller family did not die from complicated medical ailments or unknown diseases; they died because they did not have basic necessities. The environment that Isabella Miller and her children were forced to live in made them vulnerable to illness, which was only compounded by their nebulous political status; they were no longer enslaved but still unable to be independently mobile and to make their own choices. They were refugees and forced to live in unhealthy environments. During the war years, the military did not create a policy that responded to their medical needs or a program that provided them with resources; the only support promised was not realized….The Miller family did not experience liberation from chattel slavery as a jubilee, but rather as a continuous process of displacement, deprivation, and ultimately death.”
This is not the story of emancipation we typically hear – this is no day of jubil-o. Enslaved people knew that the war was their opportunity for freedom. They saw the blue uniforms of the Union soldiers – the very soldiers who sometimes sang about John Brown’s Body as they marched – as their saviors. They ‘voted with their feet,’ claiming freedom for themselves by leaving plantations and going to that army of saviors. In return, though, those people were labeled ‘contraband of war,’ rewarded with a kind of purgatory-status in which they weren’t quite free and weren’t quite slave. And while some, like Emma and Edward Whitehurst, made it out of this uncomfortable status to start their lives as freedpeople and citizens of the United States, others, like the Millers and many thousands like them, were neglected, starved, and left to the mercies of the exposure and disease. And many, many, many refugees found not freedom, but suffering, loss, and death.
[Discussion! Refugee crisis? Who freed the slaves – Lincoln, or free people themselves? Should we hate Abe Lincoln? Was Abe the ‘Great Emancipator’? I neglected women contraband badly, but read Thavolia Glymph’s work, esp. chapter “This Species of Property” ]
Bibliography:
Jim Downs, Sick From Freedom: African American Illness and Suffering During the Civil War and Reconstruction (Oxford: Oxford University Press, 2012).
Amy Murrell Taylor, Embattled Freedom: Journeys through the Civil War’s Slave Refugee Camps (Chapel Hill: University of North Carolina, 2018)
Allen Guelzo, Lincoln’s Emancipation Proclamation: The End of Slavery in America (New York: Simon and Schuster, 2004).
Adam Goodheart, 1861: The Civil War Awakening (New York: Vintage Books, 2011).
Adam Goodheart, “How Slavery Ended in America,” The New York Times Disunion
The Life and Public Letters of Major General Benjamin Butler, The Hero of New Orleans
Stephanie McCurry, The Women’s War: Fighting and Surviving the American Civil War (Cambridge: The Belknap Press of Harvard University Press, 2019)
The Declaration of Paris, 1856
Lincoln’s First Inaugural Address, March 4, 1861
0 Comments