Modified Fangjihuangqi Decoction (MFJHQD) is a formula with clear clinical efficacy for relieving heart failure (HF). However, its potential mechanisms and active substances on HF are rarely reported. In this study, ultra-performance liquid chromatography-quadrupole-time of flight-mass spectrometry was used for identifying chemical composition of MFJHQD and compounds migrating to blood. Overall, 239 compounds were identified in MFJHQD, while 79 prototype compounds and 41 metabolites were recognized in serum. Network pharmacology analysis showed that MFJHQD may modulate oxidative stress to treat HF. Subsequently, an oxidative stress model was established using tert-butyl hydroperoxide (tBHP) stimulation, and a phenotype-anchored hierarchical strategy (the formula, the medicinal herbs, and the representative compounds) was employed to identify active substances against mitochondrial oxidative damage. Experimental studies demonstrated that MFJHQD could alleviate tBHP-induced mitochondrial oxidative injury in cardiomyocytes. Among medicinal herbs of MFJHQD, Stephaniae Tetrandrae Radix, Glycyrrhizae Radix et Rhizoma, Salviae Miltiorrhizae Radix et Rhizoma, and Descurainiae Semen exhibited great activity in resisting mitochondrial oxidative damage. Tetrandrine, fangchinoline, calycosin 7-O-glucoside, isoliquiritigenin, caffeic acid, tanshinone IIA, cryptotanshinone, and quercetin-3-O-β-D-glucose-7-O-β-D-gentiobioside exhibited significant activity against mitochondrial oxidative injury. This work provides a transferable framework for investigating complex herbal formulas, and reveals the active substances in MFJHQD that alleviate mitochondrial oxidative damage, laying a foundation for its further development.