Rhodiola sacra (Prain ex Raym.-Hamet) S. H. Fu (R. sacra), a renowned medicinal Tibetan herb, is widely used in Asia and Europe for alleviating altitude sickness and reducing fatigue. However, the chemical heterogeneity across different tissues of R. sacra remains underexplored, which hinders its quality control and authentication. In this study, an integrated strategy was developed by combining non-targeted metabolomics based on ultra-high-performance liquid chromatography-Orbitrap mass spectrometry (UHPLC-Orbitrap MS) with targeted chemometrics using ultra-high-performance liquid chromatography coupled to hybrid triple quadrupole-linear ion trap mass spectrometry (UHPLC-QTRAP MS/MS). This approach was employed to comprehensively characterize the chemical constituents of R. sacra and screen for discriminatory chemical markers between the roots and aerial parts. A total of 42 metabolites were identified, including 13 phenolic acids, 13 flavonoids, 7 tannins, 2 phenylethanoid glycosides, and 7 others. Multivariate statistical analysis revealed distinct metabolic differences between the roots and aerial parts. Subsequently, six discriminatory chemical markers were selected, including epicatechin gallate, β-glucogallin, citric acid, epigallocatechin, gallic acid, and salidroside. Quantitative analysis confirmed that these chemical markers were predominantly enriched in the roots, with concentrations tens to hundreds of times higher than those in the stems and leaves. These findings provide a scientific basis for validating the root as the primary medicinal part and enhance our understanding of the tissue-specific chemical composition of R. sacra. Furthermore, these compounds serve as robust chemical markers for distinguishing roots from aerial parts, establishing a reliable framework for the quality evaluation and sustainable conservation of this precious alpine resource.