Javadoc linking to other API

I am generating Javadoc for my project and I want to link to the APIs of other projects that we use. I am doing the following: (I have tried with and without the packagelistURL and various versions of the API even though we are using 3.1)

<target name="javadoc" depends="initLocal">
    <echo>Generating javadoc</echo>
    <javadoc packagenames="*" sourcepath="${srcdir}" destdir="${docdir}" access="public" verbose="false">
        <link offline="false" href="http://docs.oracle.com/javase/6/docs/api/" />
        <link offline="false" href="http://commons.apache.org/proper/commons-lang/javadocs/api-3.1/" packagelistURL="http://commons.apache.org/proper/commons-lang/javadocs/api-3.1/package-list"/>
    </javadoc>
</target>

I can see that it is successfully linking to the Java SE6 API but I can't seem to get it to use the commons-lang API. When I run the build I see the following:

[javadoc] C:\path\to\my\project\myClass.java:6: package org.apache.commons.lang3.builder does not exist
[javadoc] import org.apache.commons.lang3.builder.ReflectionToStringBuilder;
[javadoc]                                        ^
[javadoc] C:\path\to\my\project\myClass.java:7: package org.apache.commons.lang3.builder does not exist
[javadoc] import org.apache.commons.lang3.builder.ToStringStyle;

Which seems a bit odd as the API link works and I can see both of those classes in the API. The project compiles correctly and the commons-lang jar is on the build path of the project.

Any ideas on what I am doing wrong here?