How to run concurrent Python code with asyncio

When I first tried to make concurrent HTTP requests with Python's asyncio, I completely messed it up in a hard-to-notice way. I applied the patterns I was used to from C# and made working code, but the requests ended up running one after the other without error. Here's what I did wrong, and how to do it correctly.

The wrong way

This might look like it will work, but just creating an asyncio coroutine does not actually start running it! The requests will start when they are awaited and will be made one at a time.

import asyncio
import logging.config
import sys
import httpx
import logging


logger = logging.getLogger(__name__)


async def request_and_log(client: httpx.AsyncClient, url: str) -> httpx.Response:
    logger.info("Starting request to %s", url)
    response = await client.get(url)
    logger.info("Finished request to %s", url)
    return response


async def main():
    async with httpx.AsyncClient() as client:
        js_request = request_and_log(client, "https://www.joelsleppy.com")
        st_request = request_and_log(client, "https://www.sleppytech.com")

        js_response = await js_request
        st_response = await st_request

        logger.info(
            "www.joelsleppy.com: %d",
            js_response.raise_for_status().status_code,
        )
        logger.info(
            "www.sleppytech.com: %d",
            st_response.raise_for_status().status_code,
        )


if __name__ == "__main__":
    logging.basicConfig(
        stream=sys.stdout,
        level=logging.INFO,
        format="%(asctime)s %(name)s %(message)s",
    )
    asyncio.run(main())

The log output of this is:

$ python incorrect.py
2024-12-30 10:32:28,183 __main__ Starting request to https://www.joelsleppy.com
2024-12-30 10:32:28,516 httpx HTTP Request: GET https://www.joelsleppy.com "HTTP/1.1 200 OK"
2024-12-30 10:32:28,554 __main__ Finished request to https://www.joelsleppy.com
2024-12-30 10:32:28,554 __main__ Starting request to https://www.sleppytech.com
2024-12-30 10:32:28,774 httpx HTTP Request: GET https://www.sleppytech.com "HTTP/1.1 200 OK"
2024-12-30 10:32:28,775 __main__ Finished request to https://www.sleppytech.com
2024-12-30 10:32:28,775 __main__ www.joelsleppy.com: 200
2024-12-30 10:32:28,775 __main__ www.sleppytech.com: 200

The right way

There are many different ways to achieve this, but the high-level way (and the one I'd recommend using until you find it won't work for your use case) is an asyncio.TaskGroup:

import asyncio
import logging.config
import sys
import httpx
import logging


logger = logging.getLogger(__name__)


async def request_and_log(client: httpx.AsyncClient, url: str) -> httpx.Response:
    logger.info("Starting request to %s", url)
    response = await client.get(url)
    logger.info("Finished request to %s", url)
    return response


async def main():
    async with httpx.AsyncClient() as client:
        async with asyncio.TaskGroup() as tg:
            js_request = tg.create_task(
                request_and_log(client, "https://www.joelsleppy.com")
            )
            st_request = tg.create_task(
                request_and_log(client, "https://www.sleppytech.com")
            )

        js_response = js_request.result()
        st_response = st_request.result()

        logger.info(
            "www.joelsleppy.com: %d",
            js_response.raise_for_status().status_code,
        )
        logger.info(
            "www.sleppytech.com: %d",
            st_response.raise_for_status().status_code,
        )


if __name__ == "__main__":
    logging.basicConfig(
        stream=sys.stdout,
        level=logging.INFO,
        format="%(asctime)s %(name)s %(message)s",
    )
    asyncio.run(main())

It's regrettably more verbose, but it actually runs the requests concurrently:

$ python correct.py
2024-12-30 10:35:18,162 __main__ Starting request to https://www.joelsleppy.com
2024-12-30 10:35:18,164 __main__ Starting request to https://www.sleppytech.com
2024-12-30 10:35:18,511 httpx HTTP Request: GET https://www.joelsleppy.com "HTTP/1.1 200 OK"
2024-12-30 10:35:18,513 httpx HTTP Request: GET https://www.sleppytech.com "HTTP/1.1 200 OK"
2024-12-30 10:35:18,514 __main__ Finished request to https://www.sleppytech.com
2024-12-30 10:35:18,547 __main__ Finished request to https://www.joelsleppy.com
2024-12-30 10:35:18,547 __main__ www.joelsleppy.com: 200
2024-12-30 10:35:18,548 __main__ www.sleppytech.com: 200

I hope this saves you some trouble!