The increasing demands on higher density, lower cost, higher speed, better endurance and longer retention has push flash memory technology, which is predominant and the driving force of the semiconductor nonvolatile memory market in recent years, to the position facing great challenges. However, the conventional flash memory technology using continuous highly doped polysilicon as floating gate, which is the most common in today's commercial market, can't satisfy these demands, with the transistor size continuously scaling down beyond 32 nm. Nanocrystal floating gate flash memory and SONOS-type flash memory are considered among the most promising approaches to extend scalability and performance improvement for next generation flash memory. This dissertation addresses the issues that have big effects on nanocrystal floating gate flash memory and SONOS-type flash memory performances. New device structures and new material compatible to CMOS flow are proposed and demonstrated as potential solutions for further device performance improvement. First, the effect of nanocrystal-high k dielectric interface quality on nanocrystal flash memory performance is studied. By using germanium-silicon core-shell nanocrystals or ruthenium nanocrystals buried in HfO2 as charge storage nodes, high interface quality has been achieved, leading to promising memory device performance. Next, another crucial challenge for nanocrystal flash memory on how to deposit uniformly distributed nanocrystal matrix in good shape and size control with high density is discussed. Using protein GroEL to obtain well ordered high density nanocrystal pattern, a flash memory device with Ni nanocrystals buried in HfO2 is demonstrated. For this technique, the nanocrystal size is restricted to the GroEL's central cavity size and the density is limited by protein template. To overcome this limitation, a novel method using self-assembled Co-SiO2 nanocrystals as charge storage nodes is demonstrated. Separated by thin SiO2, these nanocrystals can form close packed form to achieve ultrahigh density. Finally, charge trapping layer band engineering is proposed for SONOS-type memory for better memory performance. By manipulating the pulse ratio of Hf and Al precursor during ALD deposition, the band diagram of Hf[subscript x]Al[subscript y]O charge trapping layer is optimized to have a Hf : Al ratio 3:1 at bottom and 1:3 at the top, leading to better trade-off between programming and retention for the of memory device.