We present accurate time delays for the quadruply imaged quasar HE 0435-1223. The delays were measured from 575 independent photometric points obtained in the R-band between January 2004 and March 2010. With seven years of data, we clearly show that quasar image A is affected by strong microlensing variations and that the time delays are best expressed relative to quasar image B. We measured ΔtBC = 7.8 ± 0.8 days, ΔtBD = -6.5 ± 0.7 days and ΔtCD = -14.3 ± 0.8 days. We spacially deconvolved HST NICMOS2 F160W images to derive accurate astrometry of the quasar images and to infer the light profile of the lensing galaxy. We combined these images with a stellar population fitting of a deep VLT spectrum of the lensing galaxy to estimate the baryonic fraction, fb, in the Einstein radius. We measured fb = 0.65-0.10+0.13 if the lensing galaxy has a Salpeter IMF and fb = 0.45-0.07+0.04 if it has a Kroupa IMF. The spectrum also allowed us to estimate the velocity dispersion of the lensing galaxy, σap = 222 ± 34 km s-1. We used fb and σap to constrain an analytical model of the lensing galaxy composed of an Hernquist plus generalized NFW profile. We solved the Jeans equations numerically for the model and explored the parameter space under the additional requirement that the model must predict the correct astrometry for the quasar images. Given the current error bars on fb and σap, we did not constrain H0 yet with high accuracy, i.e., we found a broad range of models with χ2 < 1. However, narrowing this range is possible, provided a better velocity dispersion measurement becomes available. In addition, increasing the depth of the current HST imaging data of HE 0435-1223 will allow us to combine ourconstraints with lens reconstruction techniques that make use of the full Einstein ring that is visible in this object. Based on observations made with the 1.2 m Euler Swiss Telescope, the 1.5 m telescope of Maidanak Observatory in Uzbekistan, and with the 1.2 m Mercator Telescope, operated on the island of La Palma by the Flemish Community, at the Spanish Observatorio del Roque de los Muchachos of the Instituto de Astrofísica de Canarias. The NASA/ESA Hubble Space Telescope data was obtained from the data archive at the Space Telescope Science Institute, which is operated by AURA, the Association of Universities for Research in Astronomy, Inc., under NASA contract NAS-5-26555.Light curves are only available at the CDS via anonymous ftp to cdsarc.u-strasbg.fr (130.79.128.5) or via http://cdsarc.u-strasbg.fr/viz-bin/qcat?J/A+A/536/A53