Repository navigation
NodeJS native modules made difficult by distribution packages. #21897
Description
Activity
/cc @nodejs/tsc. IMO, we need to have stronger guidelines about ABI compatibility for vendors and distributors shipping Node.js. If something advertises itself as "Node.js 8.11.1" it should be compatible with the official Node.js 8.11.1.
Reacted by F. Hinkelmann, Benjamin Gruenbaum, Elana Hashman, Tobias Nießen and Timo Tijhof@ofrobots I remember talking about this in a TSC meeting before the Node 10 release (with @rvagg I think)… we knew this was coming. :/
In retrospect, maybe we should have had different
NODE_MODULE_VERSIONvalues for the two OpenSSL lines we support. Would that make sense as a pattern for the future?Reacted by Benjamin Gruenbaum@addaleax mangling the
NODE_MODULE_VERSIONshould work, yes, but this multiplies the number of packages that need to be distributed. As of today, grpc is shipping with 123 precompiled binaries for a big matrix combination of runtimes (node vs electron), platform (Windows, Linux, MacOS), architecture, and runtime versions, and we have a fairly convoluted and complex build script to generate all of these. Adding yet another dimension to this matrix doesn't make me feel warm and fuzzy, especially since it would now mean that the build script needs to acquire two different "versions" of the same node version, potentially using two different sources.I wouldn't be looking forward to this :-)
OpenSSL is probably the main suspect where the ABI between different distributors could end up differing; but there could be other environmental differences, e.g. choice of the C++ compiler, C++ std library or build flags that that make a non-official release potentially incompatible with official releases.
I think having different
NODE_MODULE_VERSIONvalues would have help this specific case of the two OpenSSL release lines, but I think there is a general problem if the ABI ends up being different in other ways.I guess my question is: Is it a problem that distributor ships an ABI incompatible version of Node that uses the same
NODE_MODULE_VERSION? I think it would good if the Node.js project could build consensus on this question, and then publish clear guidelines for vendors.My personal opinion is that ABI incompatibility fragments the ecosystem and pushes the cost of figuring out compatibility to our users. We should document clear guidelines about ABI compatibility for distributors. Native module authors now also have to worry about shipping multiple versions of the binaries for the same Node version, as @nicolasnoble points out.
If it is not compatible with Node, it is not Node.
Reacted by F. HinkelmannI discovered another, fairly edge case package: termux's nodejs package for Android.
cc @nodejs/delivery-channels
Open-source distributions implicitely assume users of Node will always compile addons and not install untrusted binaries when it can be avoided. Maybe checking for NODE_MODULE_VERSION is not enough, and shared libraries against which Node has been compiled by distributor should also be taken into account ?
@nicolasnoble my point was to make sure that what decision goes for openssl goes also for uv, zlib, openssl, c-ares, nghttp2, http-parser, icu, should multiple versions of one of these deps be supported.
Right, which would in effect make prebuilt binaries impossible to ship for anything but the official runtimes available on the website. This is an important trade-off to consider.
Maybe N-API is the solution, but i'm not even sure of that.
I don't think that N-API does anything about transitive dependencies such as openssl or libuv, no.
It is really important to emphasize the fact that this kind of problem is a general problem of open-source distributions, not particularly related to Node. There is no quick fix and the situation with Node ABI is not as simple as it looks.
The simplest solution is to fallback to building the addon from source - because open-source distributions make that really easy to achieve. sharp proposes this.
Alternatives are:- working around a c++ addon or relying on a general addon already known to work (node-ffi ?)
- distribute the c++ addon in the open-source distribution itself. Depending on the addon, it is either very quick (because popular) or not. Summoning "at nodejs/delivery-channels" with a clear request might help.
- flatpak, docker, etc... which are generic and working solutions
36 remaining items
@OlafvdSpek first, this is a really unfair comparison. Vcpkg is for C++ stuff, tailored around it, whereas native extensions in nodejs are C++ code inserted into a nodejs environment.
Sure, but building C++ code shouldn't be an unsolvable problem.
People who are installing vcpkg packages are usually C++ developer, with a C++ compiler installed, and might know how to deal with C++ issue. Nodejs users aren't necessarily having a C++ compiler installed even. And I'm not even talking about the app deployment process that are inherently different between the two.
vcpkg doesn't do app deployment at all.
Of course the goal is for building to be an 'implementation' detail, the end-developer shouldn't really be affected.And they would be affected, if only by the shifting requirement in their environment. All of a sudden, you'd need a compiler in your production environment to deploy.
Building could be done in a build / dev environment.. assuming it matches the production environment.
Or it could be done by the package provider, assuming matching environments are available.
Or it could be done by the package provider, assuming matching environments are available.
And we just went full circle, as this is exactly the topic here initially :-)
@nicolasnoble Matching environment. So not one generic Linux build, but a unique one for each platform.
Which is what we're currently discussing about, yes. The notion of having node modules tags specific to each nodejs distribution for instance, in addition to the existing ones of platform, cpu architecture, libc version, etc.
So not one generic Linux build, but a unique one for each platform.
There are hundreds of Linux distributions out there, but I don't imagine you're suggesting developers should complete a build for each and every one. I want to do better than just providing support for the popular distros. Even within each "platform" (e.g. Debian-like platforms) different releases have wildly different ABI support---the Ubuntu LTS releases look nothing like the Debian stable releases.
I maintain the toolchain for building portable native dependencies for Python. The reason building native dependencies is hard is because Linux build environments are extremely divergent. Without bundling a maximal set of dependencies with the source to build, users inevitably miss some tool or dependency. Taking this to its logical end, folks end up shipping something like a Docker artifact. While this may be appropriate in order to ship a common build environment for developers, IMO it is not appropriate for shipping software to end users. It's our job as distributors to make this easy.
The manylinux policy solves this for the Python ecosystem by defining a highly supported set of core GLIBC symbols that can be assumed to be on any system, and requires developers to build their native projects against those ABIs. Dynamic linking of dependencies is handled by the auditwheel tool which vendors the system dependencies of a Python binary package into the binary, patching the RPATH of the Python binary to point to the vendored locations. There's no reason the node ecosystem couldn't build a similar tool; the policies could even be identical.
@ehashman I'll be a bit brutal here, but over at grpc, the manylinux policy caused us way more troubles than it solved, and I'd hate seeing the same thing happening on nodejs.
@nicolasnoble that sounds like you should file some issues and we should fix them. :)
pipis now the suggested way to install numpy and other scientific Python dependencies and they have much heavier native dependency needs than grpc does, so I'm surprised to hear this. To be fair, the documentation isn't great.No, the API/ABI you've settled on is way too old and painful to deal with properly, there's quite no fixing it, since running any modern piece of code is near impossible. This TODO is basically the reason our Python support is fairly bad. It's costing us more maintenance time than anything due to the special treatment of libraries here, and due to the horribly ancient glibc we have available. The rest of the codebase has a lot of logic to deal with various API levels from the glibc, and we just can't do such things with manylinux, and we have zero options out of it.
At least with the current way nodejs does things, we can still do a lot of various work to cater for many of the distributions we want to support.
If anything, the deal with Python's manylinux comes from a good idea, but it's way too inflexible, and I wish we could opt out of it. That'd be my litmus test for any proposal you may come up with for nodejs: opt in for the average developer, opt out for the advanced ones.
@nicolasnoble does manylinux2010 address your issues? @ehashman just announced support for it in auditwheel earlier this month; see also pypa/manylinux#179 for a larger ecosystem-wide tracking issue. Have you tried building manylinux2010 wheels of grpc, and do they address any of your concerns?
So I think "The API/ABI you've settled on is way too old" would have been an entirely reasonable bug to file against pypa/manylinux. We (the volunteers working on Python packaging infrastructure) aren't going to know what issues people have if people don't tell us. We know manylinux1 is extremely old but we don't have a sense of whether moving from CentOS 5 to CentOS 6 actually helps people or not, or whether we needed something slightly newer than CentOS 6, without feedback. Advocating for the ability to upload non-manylinux wheels to PyPI would also be a reasonable bug report - it needs some careful design (same sort of design as this ticket has been talking about, more or less) but it's certainly a thing people have wanted. (Though perhaps a manylinux newer than manylinxu2010 would solve the problem too.) So would trying to find a way to build manylinux1 wheels against a newer distribution: there's a thread about using linker tricks to adjust symbol versions (and I have a long-overdue response to that waiting for me to get some free time to rigorously look at glibc's forwards-compatibility story, but the short answer is I think it can work pretty well), I wrote an awful hack to work around CentOS 5 using a syscall ABI that recent distros are dropping to make the CentOS 5 build container continue to work, etc. I'm guessing from grpc/grpc#13949 that one of the things you'd like is to use
epoll_create1(2) in a manylinux1 wheel and deal with it returning ENOSYS on old kernels - that's certainly not out of the question to support. I really don't see why you would conclude that there's "quite no fixing it" without even asking. Please open a ticket on pypa/manylinux (feel free to Cc me) describing your issues, or start a thread on wheel-builders@, and we can follow up there.To try to move this conversation back to NodeJS - it seems likely that NodeJS should base its strategy on manylinux and take into account that the Python world has found that while manylinux works pretty well, manylinux1 isn't sufficient and that periodically producing a manylinux2010, manylinux2014-ish, etc. is necessary, and that also seems to work better than defining platform variants for every single Linux distro. Among other things, it means you only have 2-3 variants to build for in CI instead of hundreds, and the installer on each platform knows what the latest manylinux it supports is, so it can gets the most optimized / featureful binary package for that platform. More to the point about OpenSSL, the Python community has generally found that bundling an OpenSSL (through the "cryptography" wheel) works better for most use cases than depending on an OpenSSL from the platform, in part because it sidesteps the question of requiring OS upgrades to speak newer TLS versions. But if you want to depend on and re-export the platform OpenSSL, you should do something manylinuxish to define what platforms have which ones, so the client installer downloads the right native modules for the system's OpenSSL version. (And it would be wonderful if the Node and Python folks could work together on defining a shared standard! OpenSSL isn't part of the manylinux list right now, but I think you could make an argument that it should be.)
More to the point about OpenSSL, the Python community has generally found that bundling an OpenSSL (through the "cryptography" wheel) works better for most use cases than depending on an OpenSSL from the platform, in part because it sidesteps the question of requiring OS upgrades to speak newer TLS versions.
Or having a distribution-provided cryptography that uses the latest system openssl. :p
But I've mentioned earlier in this issue that the specific case of openssl could be solved by exporting openssl as part of a stdlib, so that js code doesn't need to concern itself with compiling against the implementation details of openssl. This is exactly what python does. Except that both python and nodejs internally link to openssl as well... and in the python case this is controllable, since the official precompiled binaries define their own version which cryptography can match, and distros using a different version will also provide their own version of the small handful of modules that link to openssl. The python packaging ecosystem is... more canonical about where you import a module from.
Or do as python-cryptography does to sidestep the issue, which is to build with static openssl in order to not be leaky.
There are definitively options to make this work. Of course, not everyone compiles static openssl for linking against, and nodejs explicitly documents openssl as part of the public API, so compiling in private copies of openssl routines would be defeating the purpose...
When we were creating the manylinux ABI, we talked to the developers Canopy and Anaconda, which are commercial python distributions. They have a lot of hard-earned experience shipping "works anywhere" pre-compiled Linux packages to lots of users. What they told us is that they wished they could depend on the system openssl, but in practice they found openssl's ABI just wasn't consistent enough, with e.g. a history of ABI breakages even within security bugfix releases. Maybe they're better these days? I don't know. And in any case, it's obviously the case that openssl breaks compatibility between releases like 1.1.0 and 1.1.1, and personally I'd be extremely reluctant to do anything that traps distros into continuing to support older openssl releases for an indefinite period.
Also, you might want to talk to the distros before committing to anything here; our experience with Python is that their attitude towards precompiled binary packages distributed by external indices like npm or pypi ranges between "reluctantly tolerant" and "actively hostile", and they have a history of hacking up our packaging toolchains to add their own policies when redistributing them. If the distros decide that your openssl policy doesn't match their idea of what's best for their users, it's entirely possibly they'll unilaterally "fix it".
In principle it's not too hard to avoid exposing openssl as part of your ABI, but you have to wrap your head around ELF symbol lookup, which is... counterintuitive, and less helpful than it could be. In general, when you load an extension module using
dlopen(..., RTLD_LOCAL), that extension module gets its own isolated namespace. So if two different extension modules link to different versions of openssl, that's no problem, they don't interfere with each other at all [1]. BUT, the main executable is special: any symbols that it exports, or that are exported from .so's that show up when you doldd myexecutable, are effectively LD_PRELOADed into any future extension modules that executable loads. So, ifldd nodejslists openssl, then that means every extension module has to use exactly the same version of openssl as the main executable, or else the extension will have some of the symbols it's trying to pull from its openssl binary randomly overwritten by incompatible symbols from nodejs, and that's a direct train to segfaultville.Python ships with a built-in openssl wrapper, but it's built as an extension module that ships with the main interpreter, rather than linked directly into the interpreter binary itself. That's how we can avoid making openssl part of Python's ABI. This means you can absolutely take a system Python on some crusty old distro,
pip install cryptographyinto a venv, and start using cutting-edge openssl 1.1.1 features, without anything breaking. This doesn't require static linking.BTW, the same issues apply to every other .so that's linked into the main nodejs executable, like ICU, nghttp2, etc.
[1] Well, there's one special case where they can interfere, which we also have a workaround for, but it's not relevant here so let's ignore that for now.
If anything, the deal with Python's manylinux comes from a good idea, but it's way too inflexible, and I wish we could opt out of it. That'd be my litmus test for any proposal you may come up with for nodejs: opt in for the average developer, opt out for the advanced ones.
I'm sorry to hear about your problems with grpc. As Geoff explained, we are trying to migrate to a newer baseline, and are open to more fine-grained options beyond that. Unfortunately Python packaging infrastructure gets zero funding from companies beyond the minimum needed to keep PyPI's servers running, so it's slow going. But, any "opt out" will still need to solve some technical challenges. If you have a package that only runs on certain distros, how do you describe that in your package metadata, and how does your installer make use of that metadata to avoid installing packages on systems that can't support them?
Technically, it is possible to "opt out" in a sense. Google's official Tensorflow packages on PyPI simply lie about their ABI: they're built against some recent Ubuntu, but then are manually hacked to declare that they can run on any system newer than CentOS 5. So, pip happily installs them, and then they crash. It's not great.
You might be interested in this proposal I just posted.
For all its limitations, even the initial manylinux1 has been tremendously successful. Last time I checked, ~6 months ago, Python users were downloading manylinux packages ~a million times every day, and it's completely transformed the usability of Python packaging on Linux.
This issue hasn't seen action in over 1.5 years so I'm going to go ahead and close it out. FWIW, https://bugs.launchpad.net/ubuntu/+source/nodejs/+bug/1779863 was marked as fixed and I don't think there's anything to do on our end.
As a NodeJS native module developer, we've been relying on NodeJS' ABI to be able to publish pre-compiled binary packages to ease the installation process.
We recently discovered that the Debian / Ubuntu / Arch Linux packages (and maybe more ?) aren't ABI compatible with the official NodeJS distributions, which breaks pre-built binary packages in difficult ways. More specifically, these distributions are shipping NodeJS 8 that is linked against OpenSSL 1.1. The official NodeJS distribution is linking against OpenSSL 1.0, and there has been ABI breaking changes between the two versions. Therefore, a NodeJS 8 native module built against the official runtime will fail to work properly on Debian or Arch Linux's runtime.
The package we are publishing (grpc) is affected by this, but I also managed to identify at least a second package that is also affected: uws. The initial issue was reported and (painfully) investigated over on the gRPC bug tracker: grpc/grpc-node#341
I've then filed a detailed issue on Ubuntu's issue tracker to expose the problem with a reproduction case here.
I'm not sure what would the NodeJS' stance be on this issue, hence me creating this issue here to discuss the problem. I believe that the ABI breakage from Debian and Arch Linux is an oversight and an honest mistake, but I'm not sure what the resolution should be. I know that Arch Linux has an openssl-1.0 package that the nodejs package can depend upon, but I don't think this is viable for Debian / Ubuntu.
My opinion is that at the simplest, the Node foundation should publish Vendoring Guidelines, describing what it means to ship a correct NodeJS runtime, including notes on how to properly expose ABI compatible symbols for native modules.
cc @ofrobots.