Infinite or very long loop in URL validation URL

[sshort]

Using ESAPI to validate URL with the default regex in the properties file causes some URLs to loop for a very long time, while hitting high, e.g. 30% CPU usage.

Call validator.isValidInput with the following URL:

"http://mvcrzaufyh.youtube.com.hwq19b2dycja1red.youtube.com.25sq0l9.1cslt00rcg49.youtube.com.84116406.controversialtopic.net.//6js73/Q151E3//...//Q151E3//96093547/9863552/684833.bvmi?cCsRPsbcs2zDcvjJhczcvScwcdN6jcdMt"

Example stack trace:

stack_trace.txt

[xeno6696]

This is why you don't want to use regex to "validate" URLs. I think you've discovered a worst-case input.

[xeno6696]

Also, what version of ESAPI, and could you please give us the full method call as well as the exact regex rule backed by validation.properties?

[sshort]

Agreed - I wouldn't choose to use a regex for this.
Using ESAPI 2.1.0.

Regex: ^(?:(?:https?|ftp):\\/\\/)(?:\\S+(?::\\S*)?@)?(?:(?!(?:)(?:\\.\\d{1,3}){3})(?!(?:169\\.254|192\\.168)(?:\\.\\d{1,3}){2})(?!172\\.(?:1[6-9]|2\\d|3[0-1])(?:\\.\\d{1,3}){2})(?:[1-9]\\d?|1\\d\\d|2[01]\\d|22[0-3])(?:\\.(?:1?\\d{1,2}|2[0-4]\\d|25[0-5])){2}(?:\\.(?:[1-9]\\d?|1\\d\\d|2[0-4]\\d|25[0-4]))|(?:(?:[a-z\\u00a1-\\uffff0-9]-*)*[a-z\\u00a1-\\uffff0-9]+)(?:\\.(?:[a-z\\u00a1-\\uffff0-9]-*)*[a-z\\u00a1-\\uffff0-9]+)*(?:\\.(?:[a-z\\u00a1-\\uffff]{2,})))(?::\\d{2,5})?(?:\\/\\S*)?$

[xeno6696]

I cannot reproduce this, @kwwall I ran the following unit-test in ValidatorTest:

assertTrue(instance.isValidInput("test", "http://mvcrzaufyh.youtube.com.hwq19b2dycja1red.youtube.com.25sq0l9.1cslt00rcg49.youtube.com.84116406.controversialtopic.net.//6js73/Q151E3//...//Q151E3//96093547/9863552/684833.bvmi?cCsRPsbcs2zDcvjJhczcvScwcdN6jcdMt" , "URL", 2030, false));

All tests run in less than a second.

[xeno6696]

The regex backing in the default validation.properties file is this:

^(ht|f)tp(s?)\\:\\/\\/[0-9a-zA-Z]([-.\\w]*[0-9a-zA-Z])*(:(0-9)*)*(\\/?)([a-zA-Z0-9\\-\\.\\?\\,\\:\\'\\/\\\\\\+=&%\\$#_]*)?$

[xeno6696]

Yeah @sshort the problem is the regex. Where did you get it from?

[xeno6696]

I have verified that the validation.properties file in ESAPI tag 2.1.0.1 is identical to the one used in esapi tag 2.1.0. The problem is related to the custom regex. Will close as this problem isn't due to ESAPI, but in the regex used.

[sshort]

Not sure - I inherited this project. I thought I tracked it back to ESAPI itself but I shall try it out with the default regex you mention above.

Thanks for the fast responses.

[sdey1977]

Below regular expression fails to validate the URL http://core-jenkins.scansafe.cisco.com/佐贺诺伦-^ńörén.jpg appropriately. Returns FALSE after around 30911 ms (OS -X El Capitan 10.11.5,core i7 2.2 GHz )
^(ht|f)tp(s?)\\:\\/\\/[0-9a-zA-Z]([-.\\w]*[0-9a-zA-Z])*(:(0-9)*)*(\\/?)([a-zA-Z0-9\\-\\.\\?\\,\\:\\'\\/\\\\\\+=&%\\$#_]*)?$

This is the default one from validation.properties.

Please advice.

[xeno6696]

Per the URI specification:

In local or regional contexts and with improving technology, users
might benefit from being able to use a wider range of characters;
such use is not defined by this specification. Percent-encoded
octets (Section 2.1) may be used within a URI to represent characters
outside the range of the US-ASCII coded character set if this
representation is allowed by the scheme or by the protocol element in
which the URI is referenced. Such a definition should specify the
character encoding used to map those characters to octets prior to
being percent-encoded for the URI.

The URI you've provided uses non-ascii characters and is therefore outside the bounds of RFC-3986. This isn't something we will support.

If you percent-encoded the URL, like this: http%3A%2F%2Fcore-jenkins.scansafe.cisco.com%2F%E4%BD%90%E8%B4%BA%E8%AF%BA%E4%BC%A6-%5E%C5%84%C3%B6r%C3%A9n.jpg

This would meet RFC-3986, and would be supported.

[xeno6696]

To underline that the use case is unsupported, run the following program:

package org.owasp;

import java.net.URI;

public class URITest {

    public static void main(String[] args) throws Exception {
        // TODO Auto-generated method stub
        URI uri = new URI("http://core-jenkins.scansafe.cisco.com/佐贺诺伦-^ńörén.jpg");
    }

}

You should get the following exception:

Exception in thread "main" java.net.URISyntaxException: Illegal character in path at index 44: http://core-jenkins.scansafe.cisco.com/佐贺诺伦-^ńörén.jpg
    at java.net.URI$Parser.fail(URI.java:2848)
    at java.net.URI$Parser.checkChars(URI.java:3021)
    at java.net.URI$Parser.parseHierarchical(URI.java:3105)
    at java.net.URI$Parser.parse(URI.java:3053)
    at java.net.URI.<init>(URI.java:588)
    at org.owasp.URITest.main(URITest.java:9)

[xeno6696]

I should learn to read before posting. I missed this part @sdey1977 and I'm sorry.

appropriately. Returns FALSE after around 30911 ms

The problem isn't that we're not supporting an expanded character set, it's that the URI provided is a worst-case input for the regex engine.

[xeno6696]

Added the following test case to ValidatorTest

assertFalse(instance.isValidInput("test", "http://core-jenkins.scansafe.cisco.com/佐贺诺伦-^ńörén.jpg", "URL", 100, false));

Current execution time: