The relative biological effectiveness (RBE) of neutrons in neutron beams is crucial for the clinical implementation of accelerator-based boron neutron capture therapy (BNCT) systems. The RBE was quantified by comparing the doses required to achieve a 10 % cell survival fraction (D10) between reference radiation (photons) and neutrons. However, in accelerator-based BNCT, the neutron beam includes not only neutrons but also contaminating gamma rays, making it essential to calculate the RBE of neutrons while accounting for the gamma-ray dose. The RBE of neutrons was calculated using a recently proposed method, which assumes that the interaction between neutrons and contaminating gamma rays is independent, and this was compared with the conventional method, which assumes that the interactions are not independent. These calculations were conducted in an accelerator-based BNCT system with a solid-state lithium target. A comparison was also performed by varying the representative beam parameters to validate the RBE values. Additionally, the photon isoeffective dose was implemented and compared with the RBE-weighted dose calculated from the two RBE values. Four cell lines (SAS, SCCVII, U87-MG, and NB1RGB) were used to assess the cell survival fraction (SF). The SF curves for neutrons and photons were derived using linear and linear-quadratic models, respectively, to calculate D10. For the four cell lines, the mean RBE value calculated using the conventional method was 1.9 (RBE1), while that calculated using the recent method was 2.0 (RBE2). Furthermore, when the photon isoeffective dose was calculated, it closely matched the RBE-weighted doses obtained using RBE1 and RBE2 from the four cell lines. This study also examined the impact of varying the ratio of contaminating gamma rays to neutron doses, a representative beam parameter, on RBE1 and RBE2. The RBE2 value remained independent of the ratio, whereas the RBE1 value increased with rising gamma-ray contamination. However, the two RBE values was comparable when the system was adequately designed for clinical BNCT use. Therefore, this study suggests that comparing the RBE values derived from the two different methods can confirm not only the validity of the RBE but also the representative beam parameters in accelerator-based BNCT.