In humans, exposure to early life adversity has profound implications for susceptibility to developing neuropsychiatric disorders later in life. Studies in rodents have shown that stress experienced during early postnatal life can have lasting effects on brain development. Glucocorticoids and sex steroids are produced in endocrine glands and the brain from cholesterol; these molecules bind to nuclear and membrane-associated steroid receptors. Unlike other steroids that can also be made in the brain, neurosteroids bind specifically to neurotransmitter receptors, not steroid receptors. The relationships among steroids, neurosteroids, and stress are multifaceted and not yet fully understood. However, studies demonstrating altered levels of progestogens, androgens, estrogens, glucocorticoids, and their neuroactive metabolites in both developmental and adult stress paradigms strongly suggest that these molecules may be important players in stress effects on brain circuits and behavior. In this review, we discuss the influence of developmental and adult stress on various components of the brain, including neurons, glia, and perineuronal nets, with a focus on sex steroids and neurosteroids. Gaining an enhanced understanding of how early adversity impacts the intricate systems of brain steroid and neurosteroid regulation could prove instrumental in identifying novel therapeutic targets for stress-related conditions.