IGR J00291+5934 is one of the seven accreting millisecond pulsars (AMPs) discovered so far. We report on the aperiodic timing and color analysis of its X-ray flux, using all the RXTE observations of the 2004 outburst. Flat-top noise and two harmonically related quasi-periodic oscillations, all of them at very low frequencies (0.01-0.1 Hz), were present in the power spectra during most of the outburst as well as a very high fractional variability (~50%). These properties are atypical not only for AMPs but also for neutron star low-mass X-ray binaries (LMXBs) in general. There are instead some remarkable similarities with the variability observed in black hole systems, reinforcing the connections between these two types of LMXBs, as well as some interesting differences. We note finally that the results of this paper are difficult to reconcile with interpretations where any break frequency of power density spectra scales inversely with the mass of the central object at an accuracy sufficient to distinguish between the masses of neutron stars and black holes in LMXBs.