Behavioral, neuropsychological and neuroimaging research suggest a distributed network that is recruited when we interact with letters. For the first time, we combine several letter processing tasks in a single experiment to study why letters seem to engage such disparate processing areas. Using fMRI, we investigate how the brain responds to letters using tasks that should recruit systems for letter perception, letter writing, letter copying and letter imagery. We describe a network of five cortical regions including the left fusiform gyrus, two left pre-central areas, left cuneus and the left inferior frontal gyrus that are all selectively engaged during a 1-back matching paradigm with letters. Our results suggest involvement of these regions to different extents in different tasks. However, the regions also form an integrated network such that letter perception also engages motor regions while writing recruits letter-specific visual regions as well. We suggest that this distributed network is a direct result of our sensory-motor interactions with letters.