Background The relationship of coronary artery calcium (CAC) with adverse cardiac remodeling is not well established. We aimed to study the association of CAC in middle age and change in CAC from early adulthood to middle age with left ventricular (LV) function. Methods CAC score was measured by computed tomography at CARDIA study (Coronary Artery Risk Development in Young Adults) year-15 examination and at year-25 examination (Y25) in 3043 and 3189 participants, respectively. CAC score was assessed as a continuous variable and log-transformed to account for nonlinearity. Change in CAC from year-15 examination to Y25 was evaluated as the absolute difference of log-transformed CAC from year-15 examination to Y25. LV structure and function were evaluated by echocardiography at Y25. Results At Y25, mean age was 50.1±3.6 years, 56.6% women, 52.4% black. In the multivariable analysis at Y25, higher CAC was related to higher LV mass (β=1.218; adjusted P=0.007), higher LV end-diastolic volume (β=0.811; adjusted P=0.007), higher LV end-systolic volume (β=0.350; adjusted P=0.048), higher left atrial volume (β=0.214; adjusted P=0.009), and higher E/e' ratio (β=0.059; adjusted P=0.014). CAC was measured at both year-15 examination and Y25 in 2449 individuals. Higher change in CAC score during follow-up was independently related to higher LV mass index in blacks (β=4.789; adjusted P<0.001), but not in whites (β=1.051; adjusted P=0.283). Conclusions Higher CAC in middle age is associated with higher LV mass and volumes and worse LV diastolic function. Being free of CAC from young adulthood to middle age correlates to better LV function at middle age. Higher change in CAC score during follow-up is independently related to higher LV mass index in blacks.