Abstract: Based on rigorous coupled wave analysis, the models for grating-embedded multilayer structure planar diffraction were generalized to the case of conical diffraction. The proposed method can be used to calculate the light diffracted from the grating-embedded multilayer structure with arbitrary wavelength, polarization, azimuthal angle and incident angle. For the micromachined optoacoustic sensor, the convergence performance of the diffraction efficiencies of the reflected +1st order in conical diffraction was investigated. The simulations show that using a 4m grating period and TM (Transverse Magnetic) polarization, when the numbers of harmonics M(2n+1) are 67, 69 and 71, the diffraction efficiencies of the +1st order are 28.86%, 28.84% and 28.86%, respectively. Then the displacement sensitivity of the micromachined optoacoustic sensor was optimized. When the incident angle, the azimuthal angle and the grating period are 22, 10and 1 m, respectively, compared with the +1st diffracted orders of 4 m grating period, the displacement sensitivity of the 0th and +1st orders under TE and TM polarization was nearly doubled. It can be used to accurately monitor the metal membrane displacement induced by the acoustic pressure or electrostatic actuation voltage.