Neutral molecular excitations and the lowest charge transfer (CT) states between adjacent molecules play a key role for electronic devices like organic light-emitting diodes (OLEDs) and organic solar cells. For crystalline model systems composed of perylene-based chromophores, it is demonstrated that the optical properties are arising from the interplay between neutral molecular excitations and charge transfer between adjacent molecules. These excitations are coupled via electron and hole transfer, two quantities relating directly to the width of the conduction and valence band in the crystalline phase. Based on the crystal structure determined by x-ray diffraction, density-functional theory (DFT) and Hartree-Fock are used for the calculation of the electronic states of a dimer of stacked molecules. The resulting transfer parameters for electron and hole are used in an exciton model for the coupling between Frenkel excitons and CT states. DFT is used to determine molecular deformations in excited or charged electronic configurations which are then parametrized in terms of the elongation of an effective internal vibration. A comparison between the calculated dielectric function and the observed optical spectra allows us to deduce the relative energetic position of Frenkel excitons and the CT state involving stack neighbors, a key parameter for various electronic and optoelectronic device applications. This exciton model results in excellent agreement between calculated and observed optical properties.