Repository navigation
Whatwg protocol setter only seems to support special schemes #13523
Description
Activity
- changed the title
[-]Whatwg url only seems to support special urls.[/-][+]Whatwg protocol setter only seems to support special schemes[/+]on Jun 7, 2017 /cc @nodejs/url
By spec you are not allowed to change a special-scheme URL to a non-special-scheme, or vice versa. See scheme state step 2.1.1-2. However you can change from one special scheme to another (as you have found out), or from one non-special scheme to another:
const { URL } = require('url'); const url = new URL('s3://foo.bar'); url.protocol = 's4'; url.toString(); // => s4://foo.bar
The spec change was made in whatwg/url@5533c8d. See there for more details about it.
Isn't this a bug in node's URL docs, or do they mention this special case?
Not a bug as much as not documented. The docs can be expanded.
Reopening as a request to document this then.
@TimothyGu Are you sure that's how the spec is to be interpreted? My reading of https://url.spec.whatwg.org/#scheme-state, indicates that the parse method should only return in cases where
state overrideis set. Otherwise it should just go ahead and set the scheme regardless of it being special/unspecial scheme. Am I reading that wrong? And what exactly is thestate overrideargument for? 'Pretty abstract name, that.It's worth noting that Chrome does not have this issue/behavior. It sets the scheme regardless
In practice, I would argue that silently failing to do what is very obviously the desired action here (change the protocol) is arguably the worst behavior possible. At a minimum, there should be a console warning of some sort, probably even an thrown exception. I know this is probably more appropriate for discussion on the WhatWG forum, but this just wasted an hour and a half of my time and I'm not interested in throwing any more of my time into that particular sink hole.
☹️ the parse method should only return in cases where
state overrideis setYes, and:
The
protocolattribute’s setter must basic URL parse the given value, followed by U+003A (:), with context object’s url as url and scheme start state as state override.It's worth noting that Chrome does not have this issue/behavior.
Then it is Chrome's bug for not implementing whatwg/url@5533c8d.
In practice, I would argue that silently failing to do what is very obviously the desired action here (change the protocol) is arguably the worst behavior possible.
While I'm not exactly a huge fan of silently ignoring errors, we are unfortunately in a bind here, as that is what browsers as well as the URL Standard are doing for all other attributes (for example, setting
url.portto something out of range like65536will reset the port to0in Chrome). On the other hand, the URL Standard has the responsibility to maintain compatibility betweenURLandHTMLAnchorElement, as IIRCURLis intended to be a DOM-independent drop-in replacement ofHTMLAnchorElement.You have to take the entire quote into context:
The encoding override argument is a legacy concept only relevant for HTML. The url and state override arguments are only for use by various APIs. [HTML]
It's my belief that the
[HTML]link only refers to the first sentence, while by "various APIs" they mean theURLinterface (and the getters/setters of its attributes) that is visible in JavaScript.4 remaining items
@broofa It can be gleaned from whatwg/url@d5470b8 and whatwg mailing list that initially "special scheme" are meant to be those that
-
need to handle three slashes after the scheme as two, can treat lack of slashes as relative, etc.
-
cannot have an empty host (as that would lead to reparsing issues)
-
[support] backslash replacement and the encoding override
The same e-mail mentions:
I also think we should change the API such that you cannot change anything for non-relative URLs (setters are no-ops, already largely the case). And that you cannot change the scheme from a special URL to a relative URL. Given the different handling of hosts that might lead to security issues.
I'm sure @annevk (author of the aforementioned email and spec changes), @domenic, and the WHATWG bug tracker can give more insightful explanations.
Reacted by Robert Kieffer-
I feel like I'm missing some important insight.
The main reason we have "special schemes" is because URL parsers in browsers historically (and some to this day) are more per-scheme than they are generic across all URLs. That has led to subtle things being allowed for what the URL Standard calls "special schemes" that are now hard to remove as you'd break the web.
So it's basically a special case necessary for compatibility and to be able to move everyone to generic URL parsers.
Reacted by Robert KiefferI follow this with interest. One of the complaints about node's traditional
urlAPI is that it silently ignores invalid input makes a "best effort" to do something, it would be good to have the newURLAPI do better.But even if we were to implement validation errors, we would have nowhere to show them.
URL could throw exceptions.
Its not how browsers work, but browsers discard unhandled exceptions. As @mikeal has pointed out in the discussion about Promises, node chose to diverge from browser behaviour with unhandled exceptions, something that is generally considered a strength of node, not a weakness.
If there is sufficient interest it might be worth considering adding something like
URL.isValid(input)to the URL Standard, though it might take a while before that would ship in all browsers, as only Safari has a compliant implementation thus far.I am definitely very -1 on adding validation errors that are not implemented in browsers or defined by the spec.
I'd be +1 to updating the spec if necessary.
URL could throw exceptions.
We already throw exceptions for many fatal errors ("return failure" in spec-speak), but "validation errors" denote a class of URLs that are not strictly correct but still commonly seen in the wild and reasonably well-formed. In other words, we are already stricter than
url.parseby a large margin. One example is how the WHATWG API actually parses the host, and throws an error if the IP address or Punycode encoding is malformed, etc.I think deviating from browser behavior by throwing validation errors in the URL parser is much less justifiable than Promises handling, which is implicated directly by what Node.js is and is not. However, as mentioned above, I do think adding appropriate
debuglogs could be helpful if performance impact is not too significant.Also of note is that this issue is about attribute setters, not the initial parsing step, as the conversation seems to have veered towards.
#13523 (comment) would be useful then, @annevk's suggestion.
@TimothyGu @jasnell Should this stay open? I read through the thread but unclear on whether anything actionable came out or was taken to the spec side?
- added a commit that references this issue
on Aug 11, 2018 - added a commit that references this issue
on Aug 12, 2018 - added a commit that references this issue
on Apr 16, 2025 - added a commit that references this issue
on Jul 27, 2026
Currently the new whatwg url class doesn't seem to work with any other scheme except so called 'special schemes' in the spec:
https://url.spec.whatwg.org/
E.g. this works:
But this doesn't work:
Even schemes defined here: https://www.iana.org/assignments/uri-schemes/uri-schemes.xhtml don't work.
Not that this does work: