When corresponding areas of the two eyes view dissimilar images, stable perception gives way to visual competition wherein perceptual awareness alternates between those images. Moreover, a given image can remain visually dominant for several seconds at a time even when the competing images are swapped between the eyes multiple times each second. This perceptual stability across eye swaps has led to the widespread belief that this unique form of visual competition, dubbed stimulus rivalry, is governed by eye-independent neural processes at a purely binocular stage of cortical processing. We tested this idea by investigating the influence of stimulus rivalry on the buildup of the threshold elevation aftereffect, a form of contrast adaptation thought to transpire at early cortical stages that include eye-specific neural activity. Weaker threshold elevation aftereffects were observed when the adapting image was engaged in stimulus rivalry than when it was not, indicating diminished buildup of adaptation during stimulus-rivalry suppression. We then confirmed that this reduction occurred, in part, at eye-specific neural stages by showing that suppression of an image at a given moment specifically diminished adaptation associated with the eye viewing the image at that moment. Considered together, these results imply that eye-specific neural events at early cortical processing stages contribute to stimulus rivalry. We have developed a computational model of stimulus rivalry that successfully implements this idea.