A new semi-empirical method to derive the absolute flux calibration in the near infrared (J,K,L and L' bands) is proposed. The method consists of the application of the InfraRed Flux Method (IRFM) to a selected sample of stars with accurate direst measurements of their angular diameters. The absolute fluxes in each band are obtained minimizing the difference (theta(IRFM) - theta(direct)). To limit the error in the determination of the infrared monochromatic fluxes, required to apply the IRFM, a program of broad band JHKL' photometric observations has been carried out for most of the northern stars in the sample. When applying separately the method to cool and hot stars, the obtained calibrations differ by about 8% in all the photometric bands. This implies that the fundamental effective temperatures (derived from direct measurements of the angular diameters) for cool and hot stars, and those determined from the IRFM are not on the same scale, regardless of the infrared absolute calibration adopted.