IpRestrictionModule only working for localhost

So I am using IpRestrictionModule in web.config to restrict access to a specific folder thus

<location path="lucee">
<system.webServer>
<security>
<ipSecurity allowUnlisted="false"> <!-- this line blocks everybody, except those listed below -->
<clear /> <!-- removes all upstream restrictions -->
<add ipAddress="82.69.24.78" allowed="true" /> 

</ipSecurity>
</security>

.....

Access is blocked by default for unlisted IP's, and this is working

However despite my IP being listed I can only access the folder it via http://localhost/lucee

If I access the site via the domain name, then I am still blocked.

Why would it only me access the restricted folder via localhost ?