We calculate the electronic spectrum of bilayer graphene in perpendicular magnetic fields non-perturbatively. To accomodate arbitrary displacements between the two layers, we apply a periodic gauge based on singular flux vortices of phase 2π. The resulting Hofstadter-like butterfly plots show a reduced symmetry, depending on the relative position of the two layers against each other. The split of the zero-energy relativistic Landau level differs by one order of magnitude between Bernal- and non-Bernal stacking.