Repository navigation
investigate flaky test-domain-error-types #38063
Description
- Test: test-domain-error-types
- Platform: ubuntu1804_sharedlibs_debug
- Console Output:
- Build Links: https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_debug_x64/26102/consoleText
Activity
This one has been happening pretty relentlessly. It feels superfluous to add more examples, but here's one anyway.
not ok 735 parallel/test-domain-error-types --- duration_ms: 0.913 severity: crashed exitcode: -11 stack: |- ...(Seems to only happen in debug builds but happens there a lot.)
The first time this appears to happen in v8 canary build 3 days ago: https://ci.nodejs.org/job/node-test-commit-node-v8/765/
That's a sub-task of https://ci.nodejs.org/job/node-update-v8-canary/1714/. Weirdly, the main task is green even though the subtask is red.
@nodejs/v8
The first time this shows up in an actual pull request CI seems to be https://ci.nodejs.org/job/node-test-commit-linux-containered/26102/ which is a CI run for #37991 which is surely unrelated. So...something in V8 that slipped in at some point?
(Or maybe some other change messed with garbage collection or something in a way that surfaces a bug that was masked previously?)
From #37970 (comment):
I'm fairly certain this is causing #38063 (frequent test-domain-error-types failures on debug build in CI).
Here are 8 clean debug CI runs on e0eb515 the commit immediately preceding this one in master.
- https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_debug_x64/26311/
- https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_debug_x64/26310/
- https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_debug_x64/26309/
- https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_debug_x64/26308/
- https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_debug_x64/26307/
- https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_debug_x64/26306/
- https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_debug_x64/26305/
- https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_debug_x64/26304/
But here it is failing on f0bf373, the commit in this PR:
I'm going to run it a bunch more times to confirm that wasn't a fluke.
Conversation to be continued in #38063, although I'll circle back here if I find out I'm wrong and this PR isn't the cause of that issue. I'm pretty sure it is, though.
Here are 8 more CI runs to verify that f0bf373 is indeed the change where this issue starts showing up:
- https://ci.nodejs.org/job/node-test-commit-linux-containered/26312/nodes=ubuntu1804_sharedlibs_debug_x64/
- https://ci.nodejs.org/job/node-test-commit-linux-containered/26313/nodes=ubuntu1804_sharedlibs_debug_x64/
- https://ci.nodejs.org/job/node-test-commit-linux-containered/26314/nodes=ubuntu1804_sharedlibs_debug_x64/
- https://ci.nodejs.org/job/node-test-commit-linux-containered/26315/nodes=ubuntu1804_sharedlibs_debug_x64/
- https://ci.nodejs.org/job/node-test-commit-linux-containered/26316/nodes=ubuntu1804_sharedlibs_debug_x64/
- https://ci.nodejs.org/job/node-test-commit-linux-containered/26317/nodes=ubuntu1804_sharedlibs_debug_x64/
- https://ci.nodejs.org/job/node-test-commit-linux-containered/26318/nodes=ubuntu1804_sharedlibs_debug_x64/
- https://ci.nodejs.org/job/node-test-commit-linux-containered/26319/nodes=ubuntu1804_sharedlibs_debug_x64/
Now they're all green. Starting to seem like I was wasting my time bisecting because the issue was a CI issue and not an issue with the test?
Optimistically closing.
Argh, just did a resume build on node-daily-master and it showed up again.
Failed again today: https://ci.nodejs.org/job/node-test-commit-linux-containered/27102/
This now has been consistently failing https://ci.nodejs.org/job/node-test-commit-arm-debug/ (now on the equinix-ubuntu1804_sharedlibs_container-arm64 containers)
- added a commit that references this issue
on Feb 10, 2024 - added a commit that references this issue
on Feb 19, 2024