Recently, visible light communication (VLC) networks have emerged as a promising alternative for indoor data access, due to high data rate, low implementation cost, and immunity to radio frequency (RF) interference. However, the co-existence of VLC with the RF access points as well as the dependence of VLC to room illumination compel both technologies to work in parallel and thus, to form a hybrid heterogeneous VLC/RF network. This network offers the advantages of both technologies, namely increased capacity and ubiquitous coverage. Furthermore, non-orthogonal multiple access (NOMA) is a very promising candidate technique for the next generation of wireless networks, mainly due to its increased spectrum efficiency compared to orthogonal access schemes. However, the optimal user grouping in NOMA is a combinatorial NP-complete problem, which calls for low complexity techniques. To this end, in this paper, we propose the use of coalitional game theory, where the users served by the same access point (VLC or RF) form a single coalition, while the users can switch through coalitions based on their payoff. A novel utility function is proposed that takes into account the peculiarities of the NOMA hybrid VLC/RF network. Finally, a coalition formation algorithm is presented as well as an efficient power allocation policy. Computer simulations validate the presented analysis and reveal the effectiveness of the proposed user grouping scheme compared to an opportunistic approach.