How is 'instanceof' implemented in the JVM?

It's part of the JVM instruction set, basically - there's a specific instanceof instruction. So for example, a method like this:

public static void checkString(Object x) {
    if (x instanceof String) {
        System.out.println("Foo");
    }
}

is compiled into:

public static void checkString(java.lang.Object);
  Code:
     0: aload_0
     1: instanceof    #2                  // class java/lang/String
     4: ifeq          15
     7: getstatic     #3                  // Field java/lang/System.out:Ljava/io/PrintStream;
    10: ldc           #4                  // String Foo
    12: invokevirtual #5                  // Method java/io/PrintStream.println:(Ljava/lang/String;)V
    15: return
}

(That's just the output of javap.)

The JVM specification has details of what the instruction has to do. See section 6.5 for exact details of the instruction. How it's implemented is up to the VM implementation - but one sample implementation could be:

• Check whether the first operand is null (and return false if so)
• Find the execution-time type of the object that the first operand refers to.
• Navigate up the type hierarchy (including implemented interfaces) until you can prove that the actual type either is or is not compatible with the second operand.