We focus spectral structure of the quantum Rabi model (QRM) and its asymmetric version (AQRM), which are widely studied as the most fundamental models of light-matter interactions, and their covering model. The latter model is called the non-commutative harmonic oscillator (NCHO) and has rich but mysterious arithmetic properties. Certain algebraic and geometric problems/conjectures arose from the study of the degeneracy of the spectrum and hidden symmetry for AQRM will be presented. The talk is based on the joint work with Cid Reyes-Bustos and longtime collaboration with Kazufumi Kimoto.