Async Call-Stack Reconstruction

@gvanrossum

BPO	46892
Nosy	@gvanrossum, @DinoV, @asvetlov, @1st1, @kumaraditya303, @itamaro, @mpage

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2022-03-01.22:13:38.844>
labels = ['interpreter-core', '3.11', 'type-feature', 'expert-asyncio']
title = 'Async Call-Stack Reconstruction'
updated_at = <Date 2022-03-12.09:30:23.838>
user = 'https://github.com/mpage'

bugs.python.org fields:

activity = <Date 2022-03-12.09:30:23.838>
actor = 'asvetlov'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core', 'asyncio']
creation = <Date 2022-03-01.22:13:38.844>
creator = 'mpage'
dependencies = []
files = []
hgrepos = []
issue_num = 46892
keywords = []
message_count = 5.0
messages = ['414301', '414884', '414930', '414948', '414985']
nosy_count = 7.0
nosy_names = ['gvanrossum', 'dino.viehland', 'asvetlov', 'yselivanov', 'kumaraditya', 'itamaro', 'mpage']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue46892'
versions = ['Python 3.11']

Linked PRs

Profiling tools that use the call-stack (i.e. all of them) paint an incomplete picture of what’s really going on in async-heavy codebases. They can only show the stack of the currently executing task; they miss the chain of awaitables that are transitively waiting on the current task. To remedy this, we have added support in Cinder to expose the async call-stack. This consists of the call stack for the currently executing task, followed by the chain of awaitables that are transitively reachable from the currently executing task. See below for a clarifying example.

async def f1():
  return await f2()

async def f2():
  return await asyncio.ensure_future(f3())

async def f3():
  return await f4()

async def f4():
  await asyncio.sleep(10)
  return 42

When retrieved from f4, the two different stacks (top-of-stack last) are:
sync - [f3, f4]
async - [f1, f2, f3, f4]

We’d like to merge our implementation into CPython so that other heavy users of asyncio can benefit. This will consist of a few parts:

A new limited C-API to set and retrieve the “awaiter” of an awaitable object.
Additions to PyAsyncMethods to store function pointers for setting and retrieving the awaiter on instances.
An API in managed code to retrieve the async call stack as a list of fully qualified names (i.e. <module>:<class>.<function>).

We’d like to merge our implementation into CPython

Could you provide a link first, please?

Sorry for the confusion, I'm working on a PR. I filed the BPO to gauge interest in the feature.

I've recently dabbled a bit in some new primitives for asyncio, and based on that experience I think this would be very useful.

IIRC Trio does this (presumably at considerable cost) in userland.

The idea looks interesting.
The devil in the details I guess.
I'm curious what is the memory and performance penalty.
Waiting for the PR as the discussion starting point.

pythongh-91048: Fix external inspection multi-threaded performance (p…

pythongh-91048: Revert the memory cache removal for remote debugging (p…

[3.14] pythongh-91048: Revert the memory cache removal for remote deb…

pythongh-91048: Refactor common data into context object in Modukes/_…

gh-91048: Refactor common data into context object in Modules/_remote…

pythongh-91048: Fix error path result in _remote_debugging_module (py…

pythongh-91048: Refactor and optimize remote debugging module (python…

pythongh-91048: Correct Apple platform includes for iOS. (python#134712)

pythongh-91048: Add better error messages for remote debugging for CI…

Judge a cancelled task by its state, not its cancel message

Confirm subscriptions before ready; bound TCP connects (#448)

Navigation Menu

Repository navigation

Async Call-Stack Reconstruction #91048

Linked PRs

170 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Async Call-Stack Reconstruction #91048

Description

Linked PRs

Activity

mpage commented on Mar 1, 2022

asvetlov commented on Mar 11, 2022

mpage commented on Mar 11, 2022

gvanrossum commented on Mar 11, 2022

asvetlov commented on Mar 12, 2022

170 remaining items

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions