We report on optical and near-infrared observations obtained during and after the 2004 December discovery outburst of the X-ray transient and accretion-powered millisecond pulsar IGR J00291+5934. Our observations monitored the evolution of the brightness and the spectral properties of IGR J00291+5934 during the outburst decay toward quiescence. We also present optical, near-infrared, and Chandra observations obtained during true quiescence. Photometry of the field during outburst reveals an optical and near-infrared counterpart that brightened from R~=23 to R~=17 and from K=19 to K~=16. Spectral analysis of the RIJHK broadband photometry shows excess in the near-infrared bands that may be due to synchrotron emission. The Hα emission line profile suggests the orbital inclination is ~=22°-32°. The preferred range for the reddening toward the source is 0.7<=E(B-V)<=0.9, which is equivalent to 4.06×1021 cm-2<=NH<=5.22×1021 cm-2. The Chandra observations of the pulsar in its quiescent state gave an unabsorbed 0.5-10 keV flux for the best-fitting power-law model to the source spectrum of (7.0+/-0.9)×10-14 ergs cm-2 s-1 (adopting a hydrogen column of 4.6×1021 cm-2). The fit resulted in a power-law photon index of 2.4+0.5-0.4. The (R-K)0 color observed during quiescence supports an irradiated donor star and accretion disk. We estimate a distance of 2-4 kpc toward IGR J00291+5934 by using the outburst X-ray light curve and the estimated critical X-ray luminosity necessary to keep the outer parts of the accretion disk ionized. Using the quiescent X-ray luminosity and the spin period, we constrain the magnetic field of the neutron star to be <3×108 G.