Melanoma is the deadliest form of skin cancer and presents a significant health care burden in many countries. In addition to ultraviolet radiation in sunlight, the main causal factor for melanoma, genetic factors also play an important role in melanoma susceptibility. Although genome-wide association studies have identified many single nucleotide polymorphisms associated with melanoma, little is known about the proportion of disease risk attributable to these loci and their distribution throughout the genome. Here, we investigated the genetic architecture of melanoma in 1,888 cases and 990 controls of European non-Hispanic ancestry. We estimated the overall narrow-sense heritability of melanoma to be 0.18 (P < 0.03), indicating that genetics contributes significantly to the risk of sporadically-occurring melanoma. We then demonstrated that only a small proportion of this risk is attributable to known risk variants, suggesting that much remains unknown of the role of genetics in melanoma. To investigate further the genetic architecture of melanoma, we partitioned the heritability by chromosome, minor allele frequency, and functional annotations. We showed that common genetic variation contributes significantly to melanoma risk, with a risk model defined by a handful of genomic regions rather than many risk loci distributed throughout the genome. We also demonstrated that variants affecting gene expression in skin account for a significant proportion of the heritability, and are enriched among melanoma risk loci. Finally, by incorporating skin color into our analyses, we observed both a shift in significance for melanoma-associated loci and an enrichment of expression quantitative trait loci among melanoma susceptibility variants. These findings suggest that skin color may be an important modifier of melanoma risk. We speculate that incorporating skin color and other non-genetic factors into genetic studies may allow for an improved understanding of melanoma susceptibility and guide future investigations to identify melanoma risk genes.