Abstract: In order to account explicitly for the existence of long-periodic layered structures and the strong structural relaxations in the most common binary and ternary alloys of the Bi-Sb-Te-Se system, we have developed a one-dimensional cluster expansion (CE) based on first-principles electronic structure calculations, which accounts for the Bi and Sb bilayer formation. Excellent interlayer distances are obtained with a van der Waals density functional. It is shown that a CE solely based on pair interactions is sufficient to provide an accurate description of the ground-state energies of Bi-Sb-Te-Se binary and ternary systems without making the data set of ab initio calculated structures unreasonably large. For the binary alloys A1−xQx (A=Sb, Bi; Q=Te, Se), a ternary CE yields an almost continuous series of (meta)stable structures consisting of consecutive A bilayers next to consecutive A2Q3 for 0<x<0.6. For x>0.6, the binary alloy segregates into pure Q and A2Q3. The Bi-Sb system is described by a quaternary CE and is found to be an ideal solid solution stabilized by entropic effects at T≠0 K but with an ordered structure of alternating Bi and Sb layers for x=0.5 at T=0 K. A quintuple CE is used for the ternary Bi-Sb-Te system, where stable ternary layered compounds with an arbitrary stacking of Sb2Te3, Bi2Te3, and Te-Bi-Te-Sb-Te quintuple units are found, optionally separated by mixed Bi/Sb bilayers. Electronic properties of the stable compounds were studied taking spin-orbit coupling into account.