I want to get InputStream return the values I want. So I do:
doAnswer(new Answer<Byte[]>() {
    @Override
    public Byte[] answer(InvocationOnMock invocationOnMock) throws Throwable {
        return getNextPortionOfData();
    }
}).when(inputMock).read(any(byte[].class));
private Byte[] getNextPortionOfData() { ...
Exception: java.lang.Byte; cannot be cast to java.lang.Number
Question: Why?! Why I get that exception?
                        
You're trying to return a Byte[] from the call - but InputStream.read(byte[]) returns the number of bytes read, and stores the data in the byte array referenced by the parameter.
So you'd need something like:
doAnswer(new Answer<Integer>() {
    @Override
    public Integer answer(InvocationOnMock invocationOnMock) throws Throwable {
        Byte[] bytes = getNextPortionOfData();
        // TODO: Copy the bytes into the argument byte array... and
        // check there's enough space!
        return bytes.length;            
    }
});
However, I probably wouldn't use a mock for this anyway - I'd use a fake if absolutely necessary, or a ByteArrayInputStream otherwise. I'd only use a mock for really fine-grained control, such as "what happens if my input stream of encoded text returns the first half of a character in one call, then the remainder in the next..."
                    See more on this question at Stackoverflow