Bishop John England first established St. Peter's parish in 1821. The first church building, designed by Robert Mills, was demolished and replaced in 1902 by the current structure. Designed by Frank P. Milburn, architect of the State House Dome, the church features 12 original stained-glass windows thought to be from Tiffany Studios, of 11 disciples and St. Paul. The building evokes the late Gothic style of architecture and is listed on the National Register of Historic Places. It is the church in which the late Joseph Cardinal Bernardin of Chicago was baptized and confirmed. In 1987, Pope John Paul II visited the church.