The hidden symmetry of the asymmetric quantum Rabi model (AQRM) has recently been revealed via a systematic construction of the underlying symmetry operator. Based on the AQRM result, we propose an ansatz for the general form of the symmetry operators for AQRM-related models. Applying this ansatz we obtain the symmetry operator for three models: the anisotropic AQRM, the asymmetric Rabi–Stark model (ARSM), and the anisotropic ARSM.