Relative URLs in WHATWG URL API

Version: v9.x+
Platform: all
Subsystem: url

We are on the track to slowly deprecate the non-standard url.parse() (#12168 (comment)) in favor of the new WHATWG standard-based URL API. One use case that currently cannot be migrated over from url.parse() is the handling of relative URLs.

Background

url.parse() accepts incomplete, relative URLs by filling unavailable components of a URL with null.

> url.parse('#hash')
Url {
  protocol: null,
  slashes: null,
  auth: null,
  host: null,
  port: null,
  hostname: null,
  hash: '#hash',
  search: null,
  query: null,
  pathname: null,
  path: null,
  href: '#hash' }

On the other hand, the URL constructor guarantees that all URL objects are fully complete and valid URLs, which means that it throws an exception in case of relative URLs:

> new URL('#hash')
TypeError [ERR_INVALID_URL]: Invalid URL: #hash
    at Object.onParseError (internal/url.js:92:17)
    at parse (internal/url.js:101:11)
    at new URL (internal/url.js:184:5)
    at repl:1:1

WHATWG URL API does have the algorithms necessary to parse relative URLs, however, and that is activated if a base argument is provided:

> new URL('#hash', 'http://complete-url/')
URL {
  href: 'http://complete-url/#hash',
  origin: 'http://complete-url',
  protocol: 'http:',
  username: '',
  password: '',
  host: 'complete-url',
  hostname: 'complete-url',
  port: '',
  pathname: '/',
  search: '',
  searchParams: URLSearchParams {},
  hash: '#hash' }

It is not always the case that a base URL is available, though.

Possible solutions

Do nothing

What this entails is that the currently supported ability to parse relative URLs will die as url.parse() becomes deprecated.

Do not deprecate `url.parse()`; otherwise do nothing

This is the most obvious actual solution, but from tickets like #12168, I don't see this as a good idea.

Add a non-standard `TolerantURL` class

This could work if we trick the parser into believing we have a legitimate URL, except there are many conditionals in the URL parser algorithm that provide ad-hoc compatibility fixes with legacy implementations. We would have to make a set of opinionated assumptions about the nature of the URL, such as the URL's scheme.

In addition to parsing, the setters will have awkward semantics. Consider the following:

// Case 1
const relativeURL = new TolerantURL('#hash');
console.log(relativeURL.href);
  // Prints #hash

relativeURL.protocol = 'http:';
console.log(relativeURL.href);
  // Should this print http:#hash (what url.format() does)?
  // http://#hash?
  // Or make the setter a noop and therefore just #hash?

// Case 2
// Assuming we have decided to use http scheme semantics
// for TolerantURL if one is not supplied.
// The URL parser does not allow changing special-ness of
// scheme through the protocol setter.
const relativeURL = new TolerantURL('//username:password@host/');
relativeURL.protocol = 'abc';
console.log(relativeURL.href);
  // Should this print abc://username:password@host/?
  // Or //username:password@host/?

// Compare:
const absoluteURL = new URL('http://username:password@host/');
absoluteURL.protocol = 'abc';
console.log(absoluteURL.href);
  // Prints http://username:password@host/

Something else that's better than what I thought of above...

I do not suspect that we will be able to actually deprecate url.parse() for quite some time. For the foreseeable future (at least through 10.x) the two implementations will coexist side-by-side.

I vote for "do nothing", at least until it's clear what the use cases for these base-less relative URLs are and how people use them.

If we find out people use them a lot, I think a better solution would be (preferably in user-land) a "RelativeURL" class, not "TolerantURL", which only has pathname/search/searchParams/hash/toString(). Someone would have to specify how this works, but maybe as a first-pass it could have an internal real-URL and parse against https://example.com/, then just re-expose the pathname/search/searchParams/hash.

@domenic

@domenic it's worth pointing out that working with relative URLS is extremely common in Node - the most common use case I can think of is when an incoming HTTP request arrives - you get a relative URL under request.url which you typically url.parse.

I think it would be a shame to keep two URL APIs just for relative URLs. It would be really nice if URL supported relative URLs somehow - how set in store is the spec at this point regarding that?

It just doesn't make sense to have a single API for both relative and absolute URLs---the components and parsing rules are far too different depending on the base used for the rest of the API to make any sense. So I'm pretty sure the spec isn't going to change, just because there's no underlying model that makes sense.

The best you can do if you want to use one API is to make up a base URL.

@domenic

@domenic I realize that this is a hard problem, and one I do not understand very well - but I think having a base URL in Node but not in the browser could cause a lot of incompatibility when people expect code using the same spec to run the same way on both platforms.

I think we can only change URL to add example.com or something similar as a default if browsers do. If that isn't practical - then Node should stop calling url.parse a legacy API and embrace using it for relative URLs.

Which can be done, of course, using the information in an HTTP request (in the case of request.url) so I'm not overly concerned with that particular case.

The key challenge, of course, is that without a base, it's impossible to say for sure which rules to apply to the relative bits. We either must provide a base or we must provide an equivalent context in order to properly handle the URL. Otherwise the parsing will be best guess at best.

A compromise might be adding request.parsedURL() or similar (a function to show that it's expensive) which uses the request info + the request.url relative URL string to return a properly parsed URL instance with the base constructed from the request info.

@jasnell

@jasnell

Which can be done, of course, using the information in an HTTP request (in the case of request.url) so I'm not overly concerned with that particular case.

How? An HTTP server is not aware of its host name, may have several dns host names or none. In fact, I'd argue that the end server should not be concerned with what hostname it's using.

It can make a best guess using the protocol and host header, both of which may be modified of course, but it provides enough context to provide a base URL when parsing the request URI.

I'll not claim to know which, if any, of the 3 suggested solutions are best, but I just wanted to add a few things to the discussion:

I've seen incoming HTTP requests to Node servers that don't have a Host header in the wild a few times. If it's been striped by a proxy or if the client haven't provided it in the first place I don't know, but you can't rely on this header.

The 2nd issue is knowing the protocol. This can be inferred by looking at the request.socket.encrypted boolean, but this is not an exact science as the TLS might have been terminated in a load balancer.

Bottom line is that we can only safely get the path (via request.url). Getting the rest can be worked around, but not in a nice way.

So what I'm seeing is that what people really want a partial URL parser for is to parse the origin form of the request target in an HTTP request, which is defined as:

origin-form    = absolute-path [ "?" query ]

We can introduce a URLAbsolutePath class to parse from path state for this specific issue. The semantics are fairly well-defined for this specific case, since we know the scheme is always special (HTTP or HTTPS). Beyond that (including relative, query-only, or fragment-only URLs), however, users are on their own.

hmm... I'm hesitant to introduce a new class. This could be approximated in userland fairly easily using something like:

const url = new URL(`https://localhost${absolutePath}?${query}`);

Then look at the bits of url that you care about.

@jasnell

@jasnell that doesn't look very usable on user input 😕

A userland module can make it more usable. I'd rather avoid adding a convenience class that is not part of the standard

@styfle

@styfle want to move that comment over in nodejs/web-server-frameworks#71? It would be a good starter to the conversation I wanted to have there. And I have comments but don't want to hijack this thread to make them.

The URL API appears to charged blindly down the route of strictness and standards and lost a whole bunch of the utility in the process. I'm not saying necessarily that's a bad thing, but I think it makes the case that the 2 API's should probably remain because they serve 2 different purposes.

Say I want to just grab the hash portion of a relative URL url.parse('foo#bar') works just fine, new URL('foo#bar') does not. I don't care about the hostname, or the protocol, or anything like that, I just want an easy way to flexibly parse URL's, especially if those URL's have been inputted by a user. The new API has lost a lot of utility due to the lack of input flexibility.

If I want to strictly parse full URL's, fine, new URL makes sense. If I want to perform various flexible utility actions on parsed URLs, use url.parse. The two are distinct in their function and both should remain or the URL API should be made more flexible. There's little point delegating this to third party modules when there's already code to do this that's being deprecated in favour of a new API that serves and entirely different purpose.

fix: fix parsing of relative URLs

Check this one
https://nodejs.org/dist/latest-v14.x/docs/api/all.html#http_message_url

To parse the URL into its parts:

new URL(request.url, `http://${request.headers.host}`);

Once URL object is created this way we can use all its methods and properties.
Ref. Link: https://nodejs.org/dist/latest-v14.x/docs/api/url.html#url_the_whatwg_url_api

Given that we've (a) added documentation illustrating how to better handle relative URLs with the WHAT-WG API, and (b) We've backed off the deprecation of the legacy API, I'm going to close this issue for now. There's still an argument that could be made on the standards level for more ergonomic handling of relative URLs but those discussions are better directed to the whatwg/url repository.

For the sake of completeness here is the corresponding issue in the whatwg/url repository: whatwg/url#531

I think this is very important. The WHATWG API has been designed to standardise existing browser behaviour, not to be the general URL API for platforms such as NodeJS. This thread shows that this causes issues, but it is a problem not as much with NodeJS as with limitations of the standard.

I can predict that this will cause more problems down the road (and not just in Node) as the WHATWG API is becoming more widespread and people will necessarily hack around it to make it meet their needs.

I recently completed my research on the technical part of the problem by releasing this somewhat low level library. My hope is that the community can use it as a basis for building a number of more polished URL APIs that do support relative URLs whilst maintaining compatibility with URLs as defined in the WHATWG standard. I have one attempt at such an API here (but please, come up with alternatives).
The theory behind it is solid and is (still being) written down here. Any help in motivating the WHATWG to take on the issue of relative URLs is welcome.

fix(app-vite): Correctly use WHATWG URL constructor

fix(app-vite): Fix SSR publicPath handling (#12972)

@jasnell

(b) We've backed off the deprecation of the legacy API

@jasnell has that been formally declared anywhere? It may have been and I've just missed it. Should I PR the typings to remove the deprecation notice?
https://github.com/DefinitelyTyped/DefinitelyTyped/blob/master/types/node/url.d.ts#L63
https://github.com/DefinitelyTyped/DefinitelyTyped/blob/master/types/node/v16/url.d.ts#L63

Yep, if you look here https://nodejs.org/dist/latest-v18.x/docs/api/url.html#legacy-url-api, you'll see that the old API is now explicitly marked "Legacy" rather than "Deprecated" as of Node.js 15.13.0

As of Node.js 19, url.parse() is once again Deprecated 😮‍💨

https://nodejs.org/dist/latest-v19.x/docs/api/url.html#urlparseurlstring-parsequerystring-slashesdenotehost

Removed url.parse deprecation exemption

@jasnell

@jasnell, can you point me to the documentation that explains how to handle relative URLs with the WHATWG URL API please? I have read the docs here but it is unclear to me how to handle a relative url in a library in which it is undefined what context such a relative http.ClientRequest.url is used in?

Node.js docs I used: https://nodejs.org/dist/latest/docs/api/url.html#the-whatwg-url-api

My library in question: https://github.com/sbrl/powahroot/

Navigation Menu

Repository navigation

Relative URLs in WHATWG URL API #12682

Background

Possible solutions

Do nothing

Do not deprecate `url.parse()`; otherwise do nothing

Add a non-standard `TolerantURL` class

Something else that's better than what I thought of above...

29 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Relative URLs in WHATWG URL API #12682

Description

Background

Possible solutions

Do nothing

Do not deprecate url.parse(); otherwise do nothing

Add a non-standard TolerantURL class

Something else that's better than what I thought of above...

Activity

jasnell commented on Apr 28, 2017

domenic commented on Apr 29, 2017

benjamingr commented on Apr 29, 2017

domenic commented on Apr 29, 2017

benjamingr commented on Apr 29, 2017

jasnell commented on Apr 29, 2017

domenic commented on Apr 29, 2017

benjamingr commented on Apr 29, 2017

jasnell commented on Apr 29, 2017

watson commented on Jul 17, 2017

TimothyGu commented on Jul 18, 2017

jasnell commented on Jul 18, 2017

Jessidhia commented on Jul 18, 2017

jasnell commented on Jul 18, 2017

29 remaining items

wesleytodd commented on Oct 2, 2020

phawxby commented on Dec 1, 2020

mitesh1409 commented on May 2, 2021

jasnell commented on May 7, 2021

ofhouse commented on May 7, 2021

alwinb commented on May 7, 2021

phawxby commented on Jun 13, 2022

jasnell commented on Jun 13, 2022

styfle commented on Jan 17, 2023

sbrl commented on Jun 6, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Do not deprecate `url.parse()`; otherwise do nothing

Add a non-standard `TolerantURL` class