What is the encoding used for Khmer Language

A language doesn't have an encoding. What it might have is a set of characters that are commonly used when writing in that language. Some encodings only support subsets of Unicode, but both UTF-8 and UTF-16 can encode all Unicode characters, so either will work (i.e. will allow you to represent anything in Unicode) if you have the ability to choose which to use. (If you want characters which aren't in Unicode, neither UTF-8 nor UTF-16 is going to help you.)

I would typically use UTF-8 as a default - pretty much every platform supports UTF-8, whereas UTF-16 is slightly less well supported. On the other hand, you may find that there are lots of characters used by Khmer which take two bytes to encode in UTF-16 but three bytes in UTF-8, making UTF-8 take a bit more space. You should investigate this by encoding sample documents in both encodings if space is important to you.