Skip to content

Investigate flaky test-http-agent #6133

Closed
Closed
Investigate flaky test-http-agent#6133
Labels
armIssues and PRs related to the ARM architecture.httpIssues and PRs related to the http subsystem.testIssues and PRs related to Node.js core tests and test infrastructure.

Description

@mscdex
mscdex
opened on Apr 9, 2016

Example failure on pi2-raspbian-wheezy #1
Example failure on pi2-raspbian-wheezy #2

Similar issue here as in #5938?

Output:

# TIMEOUT
#0 200
#1 200
#2 200
#3 200
#4 200
#5 200
#6 200
#7 200
#8 200
#9 200
#10 200
#11 200
#12 200
#13 200
#14 200

Activity

  1. mscdex
    added
    httpIssues and PRs related to the http subsystem.
    testIssues and PRs related to Node.js core tests and test infrastructure.
    armIssues and PRs related to the ARM architecture.
    on Apr 9, 2016

  2. mscdex commented on Apr 9, 2016

    @mscdex
    mscdex
    on Apr 9, 2016
    ContributorAuthor

    Ref: #5346

  3. Trott commented on Apr 10, 2016

    @Trott
    Trott
    on Apr 10, 2016
    Member

    While looking at this test being flaky in February, it was discovered to fail always on a connection in the low 20s. In other words, if it failed, it failed on connection 22 or 23 or 24, but never on 16 or 36 or 46 or 59 or...

    Which was very odd...

    And now it seems that we're seeing that number creep lower resulting in increased flakiness on Pi 2 devices. I have no explanation. It's really weird.

    /cc @nodejs/build @rvagg

  4. Trott commented on Apr 11, 2016

    @Trott
    Trott
    on Apr 11, 2016
    Member

    Again: https://ci.nodejs.org/job/node-test-binary-arm/1681/RUN_SUBSET=3,nodes=pi2-raspbian-wheezy/tapTestReport/test.tap-53/

    Ugh. Not sure what to do. Bringing it down to 12 or so would probably fix it, but it would seem that in another month, we'd likely be facing the same issue again.

  5. Trott commented on Apr 11, 2016

    @Trott
    Trott
    on Apr 11, 2016
    Member

    cc @nodejs/testing

  6. mscdex commented on Apr 15, 2016

    @mscdex
    mscdex
    on Apr 15, 2016
    ContributorAuthor

    Again: https://ci.nodejs.org/job/node-test-binary-arm/1727/RUN_SUBSET=2,nodes=pi2-raspbian-wheezy/console

  7. santigimeno commented on Apr 15, 2016

    @santigimeno
    santigimeno
    on Apr 15, 2016
    Member

    While looking at this test being flaky in February, it was discovered to fail always on a connection in the low 20s. In other words, if it failed, it failed on connection 22 or 23 or 24, but never on 16 or 36 or 46 or 59 or...

    @Trott Would it be worth stress testing the test with the source code we had in February, just to check whether it's a problem in the code or in the raspberry bots?

  8. Trott commented on Apr 15, 2016

    @Trott
    Trott
    on Apr 15, 2016
    Member

    @santigimeno I suppose that's worth a shot.

    And, another one: https://ci.nodejs.org/job/node-test-binary-arm/1728/RUN_SUBSET=2,nodes=pi2-raspbian-wheezy/tapTestReport/test.tap-53/

  9. Trott commented on Apr 15, 2016

    @Trott
    Trott
    on Apr 15, 2016
    Member

    And another: https://ci.nodejs.org/job/node-test-binary-arm/1723/RUN_SUBSET=2,nodes=pi2-raspbian-wheezy/console

    I suppose it's superfluous to document them all here as the test is now failing so often.

  10. Trott commented on Apr 15, 2016

    @Trott
    Trott
    on Apr 15, 2016
    Member

    Per @santigimeno's suggestion, here are three CI stress tests:

  11. Trott commented on Apr 15, 2016

    @Trott
    Trott
    on Apr 15, 2016
    Member

    And one more, also at @santigimeno's suggestion:

  12. Trott commented on Apr 15, 2016

    @Trott
    Trott
    on Apr 15, 2016
    Member

    Confounding results: Previous version of the test (that used 100 connections) failed, of course.

    But no failures on any of the other stress tests, including current master.

    Maybe its flakiness is dependent on other things going on on the network or something? I mean, it shouldn't be, right? Maybe confirm that the test is using localhost and not something odd like the machine's networked IP or (would this next one even work?) a broadcast address or something...

  13. Trott commented on Apr 16, 2016

    @Trott
    Trott
    on Apr 16, 2016
    Member

    Continues to fail with alarming frequency. Running CI stress test against master. https://ci.nodejs.org/job/node-stress-single-test/596/nodes=pi2-raspbian-wheezy/console

  14. 22 remaining items

  15. Trott commented on Apr 17, 2016

    @Trott
    Trott
    on Apr 17, 2016
    Member

    Tests confirm that 757fbac is the last good commit and b85a50b is the first bad one. Now the questions are:

    • Is the problem one of configuration with the Pi2 devices?
    • Or is the problem with the code change running on Pi2 (which is just Linux, right? and not that different from other Pi devices on which we're not seeing problems, right?)
    • Or is the problem in the test somehow?

    /cc @rvagg @cjihrig

  16. cjihrig commented on Apr 18, 2016

    @cjihrig
    cjihrig
    on Apr 18, 2016
    Contributor

    Does this only happen on a specific subset of the Pi2s?

    I'd think that the change in b85a50b would cause something to break completely, not in a flaky manner. That test also doesn't have a great track record.

  17. santigimeno commented on Apr 18, 2016

    @santigimeno
    santigimeno
    on Apr 18, 2016
    Member

    FWIW. I run 4 stress test jobs with current master, reverting the suspect commit and they all passed:

  18. Trott commented on Apr 19, 2016

    @Trott
    Trott
    on Apr 19, 2016
    Member

    @cjihrig asked:

    Does this only happen on a specific subset of the Pi2s?

    Nope, I'm afraid not, at least not as far as I've been able to tell.

  19. cjihrig commented on Apr 19, 2016

    @cjihrig
    cjihrig
    on Apr 19, 2016
    Contributor

    Trying a partial revert of b85a50b. Stress test at https://ci.nodejs.org/job/node-stress-single-test/655/ with the ADDRCONFIG flag added back.

  20. santigimeno commented on Apr 19, 2016

    @santigimeno
    santigimeno
    on Apr 19, 2016
    Member

    @cjihrig, it might be worth running multiple smaller (~100 times) jobs so it picks different pi2's, just to be sure it's fixed, as we have seen runs without failures before and maybe it's somehow dependent on the bot its running on.

  21. Trott commented on Apr 20, 2016

    @Trott
    Trott
    on Apr 20, 2016
    Member

    Smaller series of stress tests on @cjihrig's partial revert per @santigimeno's suggestion:

  22. Trott commented on Apr 20, 2016

    @Trott
    Trott
    on Apr 20, 2016
    Member

    They all came back green!

  23. cjihrig
    closed this as completedin 54dd7c3on Apr 20, 2016
  24. joelostrowski
    added a commit that references this issue on Apr 25, 2016

    net: set ADDRCONFIG DNS hint in connections

    1220f3f
  25. jasnell
    added a commit that references this issue on Apr 26, 2016

    net: set ADDRCONFIG DNS hint in connections

    aa395b7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    armIssues and PRs related to the ARM architecture.httpIssues and PRs related to the http subsystem.testIssues and PRs related to Node.js core tests and test infrastructure.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions