Reading unsigned bytes with RandomAccessFile

I have this method using a RandomAccessFile descriptor:

    public byte[] readByte(int number) {
        try {
            byte[] buffer = new byte[number];
            raf.read(buffer);
            return buffer;
        } catch (IOException e) {
            throw new ReadException("readBytes failed", e);
        }

    }

Which I am trying to use to read a binary file storing a series of unsigned bytes, yet I am getting negative values. Also tried with:

public byte readByte() {
    try {
        return (byte) raf.readUnsignedByte();
    } catch (IOException e) {
        throw new ReadException("readByte failed", e);
    }
}

public byte[] readByte(int number) {
    byte[] sArray = new byte[number];
    for (int i = 0; i < number; i++) {
        sArray[i] = readByte();
    }
    return sArray;
}

It seems to me that using readUnsignedByte() should return only positive values, and yet I end up with negative values all over the place.

And even forcing positive with:

public byte[] readByte(int number) {
    byte[] sArray = new byte[number];
    for (int i = 0; i < number; i++) {
        sArray[i] = (byte) (readByte() & 0xFF);
    }
    return sArray;
}

And still, I get negative values. Only way I can use positive values is then converting them when using them (I am using these numbers as an index for another array):

palette[sArray[j] & 0xFF];

What am I doing wrong?