Reusing cached instances

I have an application that uses a data structure Point. Let's say that overall there are 50 distinct instances of Point (meaning p1.equals(p2) == false). However during calculation loads of new instances are created, that are actually the same as already instantiated objects.

As these instances are stored this has a heavy impact on memory consumption: 50 distinct Points are represented by 500'000 instances of Point. In the data structure there is nothing that would prevent the reuse of already present instances. For that reason I created a cache:

HashMap<Point, Point> pointCache = new HashMap<>();

So I can check if the point is present and add it if it is not. This kind of cache however seems like a bit of overkill, as the key and the value are essentially the same. Furthermore I already have a map present:

HashMap<Point, Boolean> flag = new HashMap<>();

What I am curious about is: Is there a map like data structure that I could use for flag that would allow the retrieval of the key? If not is there any other data structure that I could use for the cache that would be more like a set and would allow easy checking and retrevial?

EDIT: For completeness, the Point class I am using is javafx.geometry.Point2D and therefore nothing that I can change.