Scattering of light from an anisotropic source produces linear polarization in spectral lines and in the continuum. In the outer layers of a stellar atmosphere the anisotropy of the radiation field is typically dominated by the radiation escaping away, but local horizontal fluctuations of the physical conditions may also contribute, distorting the illumination and, hence, the polarization pattern. Additionally, a magnetic field may perturb and modify the line scattering polarization signals through the Hanle effect. Here, we study such symmetry-breaking effects. We develop a method to solve the transfer of polarized radiation in a scattering atmosphere with weak horizontal fluctuations of the opacity and source functions. It comprises linearization (small opacity and Planck function fluctuations are assumed), reduction to a quasi-plane-parallel problem through harmonic analysis, and the problem's numerical solution by generalized standard techniques. We apply this method to study scattering polarization in atmospheres with horizontal fluctuations in the Planck function and opacity. We derive several very general results and constraints from considerations on the symmetries and dimensionality of the problem, and we give explicit solutions of a few illustrative problems of special interest. For example, we show (1) how the amplitudes of the fractional linear polarization signals change when considering increasingly smaller horizontal atmospheric inhomogeneities, (2) that in the presence of such inhomogeneities even a vertical magnetic field may modify the scattering line polarization, and (3) that forward scattering polarization may be produced without the need for an inclined magnetic field. These results are important for understanding the physics of the problem and as benchmarks for multidimensional radiative transfer codes.