Human activities are the primary cause of increasing soil erosion worldwide. A comparative analysis of different human activity types on soil erosion can provide detailed insights into their respective roles. This study focused on Jiangxi Province, a typical red soil hilly region, to assess the impact of human activities on soil erosion based on natural influencing factors. Human activities were classified into vegetation-based human activities (VHA), soil-based human activities (SHA), and terrain-based human activities (THA). We proposed a combined impact index and a decoupling method to calculate the separate impact index for each type. The results revealed that from 2000 to 2020: (1) VHA was the dominant type, covering about 65% of the study area, followed by SHA (20%) and THA (3%). (2) The deteriorating effects of SHA were about two times greater than those of VHA, while the alleviating effect of THA was much smaller than both deteriorating effects. This study offers a new perspective for understanding the effects of human activities on soil erosion by comparing the impacts of different types of human activities.