第03章_异步编程与并发

第03章 异步编程与并发

3.1 并发基础

3.1.1 进程、线程与协程的区别

进程（Process）：

• 操作系统资源分配的基本单位
• 每个进程有独立的内存空间
• 进程间通信需要特殊机制（IPC）
• 创建和销毁开销大

线程（Thread）：

• 进程内的执行单元
• 共享进程的内存空间
• 线程间通信简单
• 受GIL限制，CPU密集型任务无法真正并行

协程（Coroutine）：

• 用户态的轻量级线程
• 由程序控制切换
• 切换开销极小
• 适合IO密集型任务

3.1.2 GIL（全局解释器锁）

GIL是Python解释器中的互斥锁，同一时刻只有一个线程执行Python字节码。

GIL的影响：

# CPU密集型任务受GIL限制
import threading

def count(n):
    while n > 0:
        n -= 1

# 创建两个线程
t1 = threading.Thread(target=count, args=(10**7,))
t2 = threading.Thread(target=count, args=(10**7,))

# 由于GIL，两个线程不会真正并行执行
t1.start()
t2.start()
t1.join()
t2.join()

3.1.3 并发与并行的区别

并发（Concurrency）：

• 多个任务交替执行
• 单核CPU也能实现
• 适合IO密集型任务

并行（Parallelism）：

• 多个任务同时执行
• 需要多核CPU
• 适合CPU密集型任务

案例1：并发与并行对比

import threading
import multiprocessing
import time

def io_bound_task():
    """IO密集型任务"""
    time.sleep(1)
    return "IO task done"

def cpu_bound_task(n):
    """CPU密集型任务"""
    result = 0
    for i in range(n):
        result += i
    return result

# 并发执行IO任务
start = time.time()
threads = [threading.Thread(target=io_bound_task) for _ in range(4)]
for t in threads:
    t.start()
for t in threads:
    t.join()
print(f"并发IO任务耗时: {time.time() - start:.2f}秒")

# 并行执行CPU任务
start = time.time()
processes = [multiprocessing.Process(target=cpu_bound_task, args=(10**8,)) for _ in range(4)]
for p in processes:
    p.start()
for p in processes:
    p.join()
print(f"并行CPU任务耗时: {time.time() - start:.2f}秒")

3.2 threading多线程

3.2.1 Thread类基础

创建线程：

import threading

def worker(name):
    print(f"Worker {name} started")
    # 执行任务
    for i in range(5):
        print(f"Worker {name}: {i}")
    print(f"Worker {name} finished")

# 创建线程
t1 = threading.Thread(target=worker, args=("A",))
t2 = threading.Thread(target=worker, args=("B",))

# 启动线程
t1.start()
t2.start()

# 等待线程完成
t1.join()
t2.join()

3.2.2 Lock锁机制

使用Lock防止竞态条件：

import threading

counter = 0
lock = threading.Lock()

def increment():
    global counter
    for _ in range(100000):
        with lock:  # 自动获取和释放锁
            counter += 1

# 创建多个线程
threads = [threading.Thread(target=increment) for _ in range(10)]

for t in threads:
    t.start()
for t in threads:
    t.join()

print(f"Counter: {counter}")  # 应该是 1,000,000

3.2.3 Event事件

使用Event进行线程间通信：

import threading
import time

event = threading.Event()

def waiter():
    print("Waiter waiting for event...")
    event.wait()  # 阻塞直到事件被设置
    print("Waiter received event!")

def setter():
    time.sleep(3)
    print("Setter setting event...")
    event.set()

# 创建线程
t1 = threading.Thread(target=waiter)
t2 = threading.Thread(target=setter)

t1.start()
t2.start()

t1.join()
t2.join()

3.2.4 Semaphore信号量

使用Semaphore限流：

import threading
import time

semaphore = threading.Semaphore(3)  # 最多允许3个线程同时执行

def worker(name):
    with semaphore:
        print(f"Worker {name} acquired semaphore")
        time.sleep(2)
        print(f"Worker {name} released semaphore")

# 创建5个线程，但只有3个能同时执行
threads = [threading.Thread(target=worker, args=(i,)) for i in range(5)]

for t in threads:
    t.start()
for t in threads:
    t.join()

3.2.5 线程池

使用ThreadPoolExecutor：

from concurrent.futures import ThreadPoolExecutor
import time

def task(name):
    print(f"Task {name} started")
    time.sleep(1)
    print(f"Task {name} finished")
    return f"Result {name}"

# 创建线程池
with ThreadPoolExecutor(max_workers=3) as executor:
    # 提交任务
    futures = [executor.submit(task, i) for i in range(5)]
    
    # 获取结果
    for future in futures:
        result = future.result()
        print(f"Got result: {result}")

案例2：线程池下载多个文件

from concurrent.futures import ThreadPoolExecutor
import requests

def download_file(url, filename):
    """下载文件"""
    response = requests.get(url)
    with open(filename, 'wb') as f:
        f.write(response.content)
    print(f"Downloaded {filename}")
    return filename

# 文件列表
files = [
    ("https://example.com/file1.txt", "file1.txt"),
    ("https://example.com/file2.txt", "file2.txt"),
    ("https://example.com/file3.txt", "file3.txt"),
]

# 使用线程池并行下载
with ThreadPoolExecutor(max_workers=3) as executor:
    futures = [executor.submit(download_file, url, filename) for url, filename in files]
    
    for future in futures:
        try:
            result = future.result()
            print(f"Successfully downloaded: {result}")
        except Exception as e:
            print(f"Download failed: {e}")

3.3 multiprocessing多进程

3.3.1 Process类基础

创建进程：

import multiprocessing

def worker(name):
    print(f"Worker {name} started")
    for i in range(5):
        print(f"Worker {name}: {i}")
    print(f"Worker {name} finished")

if __name__ == "__main__":
    # 创建进程
    p1 = multiprocessing.Process(target=worker, args=("A",))
    p2 = multiprocessing.Process(target=worker, args=("B",))
    
    # 启动进程
    p1.start()
    p2.start()
    
    # 等待进程完成
    p1.join()
    p2.join()

3.3.2 Queue队列

使用Queue进行进程间通信：

import multiprocessing

def producer(queue):
    """生产者进程"""
    for i in range(10):
        queue.put(f"Item {i}")
        print(f"Produced: Item {i}")
    queue.put(None)  # 结束标志

def consumer(queue):
    """消费者进程"""
    while True:
        item = queue.get()
        if item is None:
            break
        print(f"Consumed: {item}")

if __name__ == "__main__":
    queue = multiprocessing.Queue()
    
    p1 = multiprocessing.Process(target=producer, args=(queue,))
    p2 = multiprocessing.Process(target=consumer, args=(queue,))
    
    p1.start()
    p2.start()
    
    p1.join()
    p2.join()

3.3.3 Pool进程池

使用Pool执行并行任务：

import multiprocessing

def square(n):
    return n * n

if __name__ == "__main__":
    # 创建进程池
    with multiprocessing.Pool(processes=4) as pool:
        # map方式
        numbers = [1, 2, 3, 4, 5]
        results = pool.map(square, numbers)
        print(results)  # [1, 4, 9, 16, 25]
        
        # apply_async方式
        futures = [pool.apply_async(square, (i,)) for i in range(10)]
        results = [f.get() for f in futures]
        print(results)  # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

3.3.4 共享内存

使用Value和Array共享数据：

import multiprocessing

def update_counter(counter):
    for _ in range(100000):
        counter.value += 1

if __name__ == "__main__":
    # 创建共享计数器
    counter = multiprocessing.Value('i', 0)
    
    processes = [multiprocessing.Process(target=update_counter, args=(counter,)) for _ in range(4)]
    
    for p in processes:
        p.start()
    for p in processes:
        p.join()
    
    print(f"Counter: {counter.value}")  # 400000

案例3：多进程并行计算

import multiprocessing
import time

def calculate_partial_sum(start, end):
    """计算部分和"""
    total = 0
    for i in range(start, end):
        total += i
    return total

if __name__ == "__main__":
    # 将计算任务分成4个部分
    n = 10**9
    chunk_size = n // 4
    
    # 使用进程池
    with multiprocessing.Pool(processes=4) as pool:
        # 提交任务
        futures = []
        for i in range(4):
            start = i * chunk_size
            end = (i + 1) * chunk_size if i < 3 else n
            futures.append(pool.apply_async(calculate_partial_sum, (start, end)))
        
        # 汇总结果
        total = sum(f.get() for f in futures)
        print(f"Total sum: {total}")

3.4 asyncio异步编程

3.4.1 事件循环

事件循环是asyncio的核心：

import asyncio

async def main():
    print("Hello")
    await asyncio.sleep(1)
    print("World")

# 获取事件循环并运行
loop = asyncio.get_event_loop()
loop.run_until_complete(main())

3.4.2 async/await语法

基本异步函数：

import asyncio

async def say_hello(name):
    print(f"Hello, {name}")
    await asyncio.sleep(1)
    print(f"Goodbye, {name}")

async def main():
    # 并发执行多个协程
    await asyncio.gather(
        say_hello("Alice"),
        say_hello("Bob"),
        say_hello("Charlie")
    )

asyncio.run(main())

3.4.3 Task任务

创建Task并发执行：

import asyncio

async def worker(name, delay):
    print(f"Worker {name} started")
    await asyncio.sleep(delay)
    print(f"Worker {name} finished after {delay}s")
    return f"Result from {name}"

async def main():
    # 创建任务
    task1 = asyncio.create_task(worker("A", 2))
    task2 = asyncio.create_task(worker("B", 1))
    
    # 等待任务完成
    result1 = await task1
    result2 = await task2
    
    print(f"Results: {result1}, {result2}")

asyncio.run(main())

3.4.4 asyncio.gather

并发执行多个协程并收集结果：

import asyncio

async def fetch_data(url):
    print(f"Fetching {url}")
    await asyncio.sleep(1)  # 模拟网络请求
    return f"Data from {url}"

async def main():
    urls = ["https://example.com", "https://google.com", "https://github.com"]
    
    # 并发获取所有数据
    results = await asyncio.gather(*[fetch_data(url) for url in urls])
    
    for result in results:
        print(result)

asyncio.run(main())

3.4.5 asyncio.wait

灵活控制任务等待：

import asyncio

async def task(delay, name):
    await asyncio.sleep(delay)
    print(f"Task {name} completed")
    return name

async def main():
    tasks = [
        asyncio.create_task(task(2, "A")),
        asyncio.create_task(task(1, "B")),
        asyncio.create_task(task(3, "C"))
    ]
    
    # 等待第一个完成的任务
    done, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED)
    
    print(f"Completed tasks: {len(done)}")
    for t in done:
        print(f"Result: {t.result()}")
    
    # 取消剩余任务
    for t in pending:
        t.cancel()

asyncio.run(main())

案例4：异步并发爬虫

import asyncio
import aiohttp

async def fetch(session, url):
    """异步获取URL内容"""
    async with session.get(url) as response:
        return await response.text()

async def main():
    urls = [
        "https://httpbin.org/get",
        "https://httpbin.org/headers",
        "https://httpbin.org/ip"
    ]
    
    # 创建会话
    async with aiohttp.ClientSession() as session:
        # 并发请求
        tasks = [fetch(session, url) for url in urls]
        results = await asyncio.gather(*tasks)
        
        for url, content in zip(urls, results):
            print(f"URL: {url}")
            print(f"Content length: {len(content)}")

asyncio.run(main())

3.5 异步IO

3.5.1 aiohttp异步HTTP客户端

基本用法：

import asyncio
import aiohttp

async def main():
    async with aiohttp.ClientSession() as session:
        # GET请求
        async with session.get('https://httpbin.org/get') as response:
            print(f"Status: {response.status}")
            print(f"Content-Type: {response.headers['content-type']}")
            data = await response.json()
            print(data)
        
        # POST请求
        payload = {'key': 'value'}
        async with session.post('https://httpbin.org/post', json=payload) as response:
            data = await response.json()
            print(data)

asyncio.run(main())

3.5.2 aiofiles异步文件操作

异步文件读写：

import asyncio
import aiofiles

async def read_file(filename):
    """异步读取文件"""
    async with aiofiles.open(filename, mode='r') as f:
        content = await f.read()
        return content

async def write_file(filename, content):
    """异步写入文件"""
    async with aiofiles.open(filename, mode='w') as f:
        await f.write(content)

async def main():
    # 写入文件
    await write_file('example.txt', 'Hello, World!\n')
    
    # 读取文件
    content = await read_file('example.txt')
    print(content)

asyncio.run(main())

3.5.3 异步上下文管理器

自定义异步上下文管理器：

import asyncio

class AsyncResource:
    def __init__(self, name):
        self.name = name
    
    async def __aenter__(self):
        print(f"Opening {self.name}")
        await asyncio.sleep(0.5)  # 模拟初始化
        return self
    
    async def __aexit__(self, exc_type, exc_val, exc_tb):
        print(f"Closing {self.name}")
        await asyncio.sleep(0.5)  # 模拟清理

async def main():
    async with AsyncResource("database") as resource:
        print(f"Using {resource.name}")

asyncio.run(main())

案例5：异步日志记录器

import asyncio
import aiofiles
from datetime import datetime

class AsyncLogger:
    def __init__(self, filename):
        self.filename = filename
    
    async def log(self, message, level="INFO"):
        """异步记录日志"""
        timestamp = datetime.now().isoformat()
        log_entry = f"[{timestamp}] [{level}] {message}\n"
        
        async with aiofiles.open(self.filename, mode='a') as f:
            await f.write(log_entry)

async def main():
    logger = AsyncLogger("app.log")
    
    # 并发记录日志
    await asyncio.gather(
        logger.log("Application started"),
        logger.log("User logged in", level="DEBUG"),
        logger.log("Error occurred", level="ERROR")
    )

asyncio.run(main())

3.6 协程高级

3.6.1 asyncio.Queue

异步队列：

import asyncio

async def producer(queue):
    """生产者"""
    for i in range(5):
        item = f"Item {i}"
        await queue.put(item)
        print(f"Produced: {item}")
        await asyncio.sleep(0.5)

async def consumer(queue):
    """消费者"""
    while True:
        item = await queue.get()
        print(f"Consumed: {item}")
        queue.task_done()
        if item == "Item 4":
            break

async def main():
    queue = asyncio.Queue(maxsize=2)  # 限制队列大小
    
    # 创建任务
    producer_task = asyncio.create_task(producer(queue))
    consumer_task = asyncio.create_task(consumer(queue))
    
    await producer_task
    await consumer_task

asyncio.run(main())

3.6.2 Semaphore限流

异步信号量：

import asyncio

async def worker(name, semaphore):
    """受限流的工作协程"""
    async with semaphore:
        print(f"Worker {name} acquired semaphore")
        await asyncio.sleep(1)
        print(f"Worker {name} released semaphore")

async def main():
    semaphore = asyncio.Semaphore(2)  # 最多2个并发
    
    # 创建5个工作协程
    tasks = [worker(i, semaphore) for i in range(5)]
    await asyncio.gather(*tasks)

asyncio.run(main())

3.6.3 超时控制

使用timeout控制执行时间：

import asyncio

async def long_running_task():
    """模拟长时间运行的任务"""
    await asyncio.sleep(5)
    return "Done"

async def main():
    try:
        # 设置超时时间
        result = await asyncio.wait_for(long_running_task(), timeout=2)
        print(f"Result: {result}")
    except asyncio.TimeoutError:
        print("Task timed out!")

asyncio.run(main())

案例6：带超时的异步请求

import asyncio
import aiohttp

async def fetch_with_timeout(session, url, timeout=5):
    """带超时的异步请求"""
    try:
        async with asyncio.timeout(timeout):
            async with session.get(url) as response:
                return await response.text()
    except asyncio.TimeoutError:
        print(f"Request to {url} timed out")
        return None

async def main():
    urls = [
        "https://httpbin.org/delay/3",  # 延迟3秒
        "https://httpbin.org/delay/10", # 延迟10秒（会超时）
        "https://httpbin.org/get"
    ]
    
    async with aiohttp.ClientSession() as session:
        tasks = [fetch_with_timeout(session, url, timeout=5) for url in urls]
        results = await asyncio.gather(*tasks)
        
        for url, result in zip(urls, results):
            if result:
                print(f"Success: {url}")
            else:
                print(f"Failed: {url}")

asyncio.run(main())

3.7 并发模式实战

3.7.1 生产者-消费者模式

异步生产者-消费者：

import asyncio
import random

async def producer(queue, name):
    """生产者协程"""
    for i in range(5):
        item = f"{name}-Item-{i}"
        await queue.put(item)
        print(f"Producer {name} put: {item}")
        await asyncio.sleep(random.uniform(0.1, 0.5))

async def consumer(queue, name):
    """消费者协程"""
    while True:
        item = await queue.get()
        print(f"Consumer {name} got: {item}")
        await asyncio.sleep(random.uniform(0.2, 0.6))
        queue.task_done()
        
        if item.endswith("-Item-4"):
            break

async def main():
    queue = asyncio.Queue()
    
    # 创建生产者和消费者
    producers = [asyncio.create_task(producer(queue, f"P{i}")) for i in range(2)]
    consumers = [asyncio.create_task(consumer(queue, f"C{i}")) for i in range(3)]
    
    # 等待生产者完成
    await asyncio.gather(*producers)
    
    # 等待队列处理完毕
    await queue.join()
    
    # 取消消费者
    for consumer_task in consumers:
        consumer_task.cancel()

asyncio.run(main())

3.7.2 扇入扇出模式

扇出：一个任务分发到多个任务

import asyncio

async def worker(task_id, data):
    """处理数据的工作协程"""
    result = data * 2
    await asyncio.sleep(0.1)
    return f"Worker {task_id}: {result}"

async def fan_out(data_list):
    """扇出模式：并行处理多个数据"""
    tasks = [worker(i, data) for i, data in enumerate(data_list)]
    results = await asyncio.gather(*tasks)
    return results

asyncio.run(fan_out([1, 2, 3, 4, 5]))  # ['Worker 0: 2', 'Worker 1: 4', ...]

扇入：多个任务合并到一个任务

import asyncio

async def fetch_source(url):
    """从不同来源获取数据"""
    await asyncio.sleep(0.5)
    return f"Data from {url}"

async def fan_in(urls):
    """扇入模式：汇总多个来源的数据"""
    tasks = [fetch_source(url) for url in urls]
    results = await asyncio.gather(*tasks)
    return "\n".join(results)

asyncio.run(fan_in(["source1", "source2", "source3"]))

3.7.3 流水线模式

流水线：多个阶段依次处理

import asyncio

async def stage1(data):
    """第一阶段：预处理"""
    await asyncio.sleep(0.2)
    return data.upper()

async def stage2(data):
    """第二阶段：转换"""
    await asyncio.sleep(0.3)
    return f"[{data}]"

async def stage3(data):
    """第三阶段：输出"""
    await asyncio.sleep(0.1)
    print(f"Final: {data}")
    return data

async def pipeline(item):
    """单个项目的流水线处理"""
    result1 = await stage1(item)
    result2 = await stage2(result1)
    result3 = await stage3(result2)
    return result3

async def main():
    items = ["task1", "task2", "task3", "task4"]
    
    # 并发执行多个流水线
    tasks = [pipeline(item) for item in items]
    await asyncio.gather(*tasks)

asyncio.run(main())

案例7：异步数据处理流水线

import asyncio

async def fetch_data(source):
    """模拟从数据源获取数据"""
    await asyncio.sleep(0.3)
    return {"source": source, "data": [1, 2, 3]}

async def process_data(raw_data):
    """处理原始数据"""
    await asyncio.sleep(0.2)
    processed = [x * 2 for x in raw_data["data"]]
    return {"source": raw_data["source"], "processed": processed}

async def save_data(processed_data):
    """保存处理后的数据"""
    await asyncio.sleep(0.1)
    print(f"Saved: {processed_data}")
    return True

async def data_pipeline(source):
    """完整的数据处理流水线"""
    raw = await fetch_data(source)
    processed = await process_data(raw)
    saved = await save_data(processed)
    return saved

async def main():
    sources = ["database", "api", "file"]
    
    # 并发执行多个流水线
    results = await asyncio.gather(*[data_pipeline(source) for source in sources])
    print(f"All pipelines completed: {all(results)}")

asyncio.run(main())

3.8 性能对比

3.8.1 IO密集型场景

场景描述：

• 大量网络请求
• 文件读写操作
• 数据库查询

推荐方案：

• asyncio + aiohttp（异步IO）
• threading（线程池）

对比测试：

import asyncio
import threading
import time
import requests
import aiohttp

# 同步版本
def sync_requests(urls):
    results = []
    for url in urls:
        results.append(requests.get(url).text)
    return results

# 线程版本
def thread_requests(urls):
    def fetch(url, results, index):
        results[index] = requests.get(url).text
    
    results = [None] * len(urls)
    threads = [threading.Thread(target=fetch, args=(urls[i], results, i)) for i in range(len(urls))]
    
    for t in threads:
        t.start()
    for t in threads:
        t.join()
    
    return results

# 异步版本
async def async_requests(urls):
    async with aiohttp.ClientSession() as session:
        async def fetch(url):
            async with session.get(url) as response:
                return await response.text()
        
        tasks = [fetch(url) for url in urls]
        return await asyncio.gather(*tasks)

# 测试
urls = ["https://httpbin.org/get"] * 10

# 同步
start = time.time()
sync_requests(urls)
print(f"Sync: {time.time() - start:.2f}s")

# 线程
start = time.time()
thread_requests(urls)
print(f"Thread: {time.time() - start:.2f}s")

# 异步
start = time.time()
asyncio.run(async_requests(urls))
print(f"Async: {time.time() - start:.2f}s")

3.8.2 CPU密集型场景

场景描述：

• 大量计算任务
• 数据处理算法
• 图像处理

推荐方案：

• multiprocessing（多进程）

对比测试：

import threading
import multiprocessing
import time

def compute_intensive(n):
    """CPU密集型计算"""
    result = 0
    for i in range(n):
        result += i ** 2
    return result

# 同步版本
def sync_compute():
    results = []
    for _ in range(4):
        results.append(compute_intensive(10**8))
    return results

# 线程版本
def thread_compute():
    results = [None] * 4
    threads = [threading.Thread(target=lambda i: results.__setitem__(i, compute_intensive(10**8)), args=(i,)) for i in range(4)]
    
    for t in threads:
        t.start()
    for t in threads:
        t.join()
    
    return results

# 多进程版本
def process_compute():
    with multiprocessing.Pool(processes=4) as pool:
        return pool.map(compute_intensive, [10**8] * 4)

# 测试
# 同步
start = time.time()
sync_compute()
print(f"Sync: {time.time() - start:.2f}s")

# 线程（受GIL限制，性能提升有限）
start = time.time()
thread_compute()
print(f"Thread: {time.time() - start:.2f}s")

# 多进程（真正并行）
start = time.time()
process_compute()
print(f"Process: {time.time() - start:.2f}s")

3.8.3 场景选型指南

小结

本章介绍了Python中的并发编程，包括：

1. 并发基础概念（进程、线程、协程、GIL）
2. threading多线程编程（Lock、Event、Semaphore、线程池）
3. multiprocessing多进程编程（Process、Queue、Pool、共享内存）
4. asyncio异步编程（事件循环、async/await、Task、gather）
5. 异步IO（aiohttp、aiofiles、异步上下文管理器）
6. 协程高级特性（Queue、Semaphore、超时控制）
7. 并发模式实战（生产者-消费者、扇入扇出、流水线）
8. 性能对比与场景选型指南

选择合适的并发方案需要根据具体场景（IO密集型vs CPU密集型）来决定，asyncio适合IO密集型任务，multiprocessing适合CPU密集型任务。