Adapted from Wikipedia:
The building was 253 feet, 3 inches in length by 200 feet, 5 inches wide and it occupied a full block, surrounded by Centre, Franklin Street, Elm (today’s Lafayette), and Leonard Streets. It initially accommodated about 300 prisoners.

The Collect Pond and Five Points on the topographical map by Egbert Viele. The Five Points intersection is where Mosco Street (marked here as Park Street) intersected with Baxter Street (formerly Orange Street) and Worth Street (formerly Anthony Street).
The building site had been created by filling in the Collect Pond, a freshwater pond that had once been the principal water source for colonial New York City. Industrialization and population density by the late 18th century resulted in the severe pollution of the Collect. It was condemned, drained and filled in by 1817. The landfill job was poorly done, and in a span of less than ten years, the ground began to subside. The resulting swampy, foul-smelling conditions had already resulted in the quick transformation of the neighborhood into a slum known as Five Points by the time construction of the prison started in 1838. The enormous, heavy masonry of Haviland’s building was built atop vertical piles of gigantic lashed hemlock tree trunks in a bid for stability, but the entire structure began to sink soon after it was opened. This damp foundation was primarily responsible for its bad reputation as being unsanitary during the decades to come.
As it also housed the city’s courts, police, and detention facilities, The Tombs’ formal title was The New York Halls of Justice and House of Detention. Some regarded it as a notable example of Egyptian Revival architecture in the U.S., but opinion varied greatly concerning its actual merit. “What is this dismal fronted pile of bastard Egyptian, like an enchanter’s palace in a melodrama?”, asked Charles Dickens in his American Notes of 1842.