Here I need to cast to double precision to integer.
Example:
obj.DayDifference = !string.IsNullOrEmpty(reader["DateDiff"].ToString()) ?
(Int32)reader["DateDiff"] : 0;
Type:
DayDifference of type int
DateDiff of type double precision in the database table.
Error:
Specified cast is not valid

What you're doing is equivalent to:
object x = 32.5;
int y = (int) x;
You can't do that - when you unbox, you have to unbox to the actual type of the value1.
So you'd need:
object x = 32.5;
int y = (int) (double) x;
The cast to double unboxes, and the cast to int converts the double to an int. You can do the same in your code:
(Int32)(Double) reader["DateDiff"] : 0;
1 Well, modulo a few things. The CLR allows you to unbox an int to a uint or to an enum type with an underlying type of int, etc.
See more on this question at Stackoverflow