We present simultaneous optical and near-infrared (IR) photometry of the millisecond pulsar PSR J1023+0038 during its low-mass X-ray binary phase. The r΄- and Ks-band light curves show rectangular, flat-bottomed dips, similar to the X-ray mode-switching (active-passive state transitions) behaviour observed previously. The cross-correlation function (CCF) of the optical and near-IR data reveals a strong, broad negative anticorrelation at negative lags, a broad positive correlation at positive lags, with a strong, positive narrow correlation superimposed. The shape of the CCF resembles the CCF of black hole X-ray binaries but the time-scales are different. The features can be explained by reprocessing and a hot accretion flow close to the neutron star's magnetospheric radius. The optical emission is dominated by the reprocessed component, whereas the near-IR emission contains the emission from plasmoids in the hot accretion flow and a reprocessed component. The rapid active-passive state transition occurs when the hot accretion flow material is channelled on to the neutron star and is expelled from its magnetosphere. During the transition the optical reprocessing component decreases resulting in the removal of a blue spectral component. The accretion of clumpy material through the magnetic barrier of the neutron star produces the observed near-IR/optical CCF and variability. The dip at negative lags corresponds to the suppression of the near-IR synchrotron component in the hot flow, whereas the broad positive correlation at positive lags is driven by the increased synchrotron emission of the outflowing plasmoids. The narrow peak in the CCF is due to the delayed reprocessed component, enhanced by the increased X-ray emission.