爬取一个静态网站的所有数据

这是一份完整的文档，对整个爬取静态网站数据的程序进行详细说明和代码组织，同时保留所有原有代码。该程序主要目标是将静态网站的所有网页数据（HTML、PDF、转换后的 Markdown 等）存入数据库，为后续使用大模型回答用户问题提供参考依据。整个项目基于 litongjava/playwright-server 开发，并利用 Java 21 的虚拟线程来提升并发爬取的性能。

项目背景

本项目的核心功能是爬取一个静态网站的所有数据，提取每个页面的 HTML 内容、PDF 文件文本以及转换为 Markdown 格式后的数据，并将这些数据存入数据库中。系统利用 Playwright 进行页面访问和渲染，结合 Java 21 的虚拟线程和线程池管理来实现高并发爬取，并采用 Jsoup、PDFBox、Flexmark 等第三方工具完成数据提取与转换。

项目整体架构与模块说明

项目主要分为以下几个模块，各模块协同完成静态网站数据爬取、转换和存储的全流程：

配置与初始化

通过 PlaywrightConfig 类在系统启动时完成 Playwright 相关资源的初始化，包括根据环境配置初始化固定数量的浏览器上下文池。同时，注册关闭钩子以确保系统退出时能够自动释放资源。

浏览器上下文池管理

利用 PlaywrightPool 类和内部包装的 PooledPage 类，实现浏览器上下文的池化管理。这样避免了频繁创建与销毁浏览器上下文的性能开销，并通过虚拟线程提高并发效率。

爬虫服务与任务调度

任务入口：由 CrawlController 提供 HTTP 接口，启动爬虫任务。
任务调度与爬虫逻辑：CrawlWebPageTask 类负责不断从 URL 任务队列中获取任务，并针对 HTML 页面和 PDF 文件分别进行处理。页面解析中利用 Jsoup 解析 DOM，提取所有有效链接，加入任务队列中保证完整爬取同一域下所有数据。
并发执行：使用 TaskExecutorUtils 中的线程池来并发执行爬虫任务，利用 Java 虚拟线程实现高效任务调度。

数据持久化

使用 WebPageDao 类进行数据存储，保存原始 HTML、转换后的 Markdown 和 PDF 文本数据到数据库中。系统采用基于雪花算法生成的唯一 ID，并通过数据库表 web_page_cache 等记录数据。

工具类与辅助功能

项目中还提供了若干辅助工具类：

MarkdownUtils：利用 FlexmarkHtmlConverter 将 HTML 转换为 Markdown 格式。
PDFUtils：借助 PDFBox 解析 PDF 文档，提取其中文本内容。
WebsiteUrlUtils：对 URL 进行域名提取、标准化和规范化处理，保证同一页面不会重复爬取。
TaskExecutorUtils：管理爬虫任务的线程池，确保任务提交和拒绝策略得当。
内部任务模型：CrawlTask 用于记录待爬取 URL 及其深度信息（后续可扩展）。

各模块代码详细说明

下面按照合理的组织顺序依次展示各个 Java 类的代码，并附加必要的解释说明。

1. 数据库表结构 SQL

下面提供了创建数据缓存、页面数据、URL 管理等相关表的 SQL 语句。这些表分别存储了网页缓存数据、目标页面数据以及待爬取 URL 任务队列。

数据库创建语句定义了用于缓存页面数据、URL 队列和目标网站页面数据的多个表。

drop table if exists web_page_cache;
CREATE TABLE "public"."web_page_cache" (
  "id" BIGINT NOT NULL PRIMARY KEY,
  "url" VARCHAR UNIQUE,
  "title" VARCHAR,
  "type" VARCHAR,
  "text" text,
  "html" text,
  "markdown" text,urlurl
  "remark" VARCHAR(256),
  "creator" VARCHAR(64) DEFAULT '',
  "create_time" TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
  "updater" VARCHAR(64) DEFAULT '',
  "update_time" TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
  "deleted" SMALLINT DEFAULT 0,
  "tenant_id" BIGINT NOT NULL DEFAULT 0
);

CREATE INDEX "web_page_cache_url" ON "web_page_cache" USING btree ("url" varchar_pattern_ops);
CREATE INDEX "web_page_cache_title" ON "web_page_cache" USING btree ("title" varchar_pattern_ops);


drop table if exists hawaii_web_page;
CREATE TABLE "public"."hawaii_web_page" (
  "id" BIGINT NOT NULL PRIMARY KEY,
  "url" VARCHAR UNIQUE,
  "title" VARCHAR,
  "type" VARCHAR,
  "text" text,
  "html" text,
  "markdown" text,
  "remark" VARCHAR(256),
  "creator" VARCHAR(64) DEFAULT '',
  "create_time" TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
  "updater" VARCHAR(64) DEFAULT '',
  "update_time" TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
  "deleted" SMALLINT DEFAULT 0,
  "tenant_id" BIGINT NOT NULL DEFAULT 0
);

CREATE INDEX "hawaii_web_page_url" ON "hawaii_web_page" USING btree ("url" varchar_pattern_ops);
CREATE INDEX "hawaii_web_page_title" ON "hawaii_web_page" USING btree ("title" varchar_pattern_ops);

drop table if exists web_page_url;
CREATE TABLE "public"."web_page_url" (
  "id" BIGINT NOT NULL PRIMARY KEY,
  "url" VARCHAR UNIQUE,
  "status" int default 0,
  "tried" int default 0,
  "remark" VARCHAR(256),
  "creator" VARCHAR(64) DEFAULT '',
  "create_time" TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
  "updater" VARCHAR(64) DEFAULT '',
  "update_time" TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
  "deleted" SMALLINT DEFAULT 0,
  "tenant_id" BIGINT NOT NULL DEFAULT 0
);

CREATE INDEX "web_page_url_url" ON "web_page_url" USING btree ("url" varchar_pattern_ops);

-- status 0 添加到任务队列 1 爬取完成 2.如何判断爬取失败

drop table if exists hawaii_kapiolani_web_page;
CREATE TABLE "public"."hawaii_kapiolani_web_page" (
  "id" BIGINT NOT NULL PRIMARY KEY,
  "url" VARCHAR UNIQUE,
  "title" VARCHAR,
  "type" VARCHAR,
  "text" text,
  "html" text,
  "markdown" text,
  "remark" VARCHAR(256),
  "creator" VARCHAR(64) DEFAULT '',
  "create_time" TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
  "updater" VARCHAR(64) DEFAULT '',
  "update_time" TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
  "deleted" SMALLINT DEFAULT 0,
  "tenant_id" BIGINT NOT NULL DEFAULT 0
);

2. 配置与初始化

在 PlaywrightConfig 中，根据当前环境启动 Playwright，并初始化浏览器池。还注册了服务关闭时自动释放资源的钩子。

package nexus.io.playwright.config;

import nexus.io.annotation.AConfiguration;
import nexus.io.annotation.Initialization;
import nexus.io.hook.HookCan;
import nexus.io.playwright.pool.PlaywrightPool;
import nexus.io.tio.utils.environment.EnvUtils;

import lombok.extern.slf4j.Slf4j;

@AConfiguration
@Slf4j
public class PlaywrightConfig {

  @Initialization
  public void config() {
    if (EnvUtils.getBoolean("playwright.enable", true)) {
      // 启动
      log.info("start init playwright");
      if (EnvUtils.isDev()) {
        PlaywrightPool.init(2);
      } else {
        int cpuCount = Runtime.getRuntime().availableProcessors();
        PlaywrightPool.init(cpuCount);
      }
      log.info("end init playwright");

      // 服务关闭时，自动关闭浏览器和 Playwright 实例
      HookCan.me().addDestroyMethod(() -> {
        PlaywrightPool.close();
      });
    }
  }
}

3. 浏览器池管理

3.1 PlaywrightPool

该类负责初始化 Playwright 实例、启动浏览器、创建固定数量的 BrowserContext，并通过 BlockingQueue 实现池化管理。同时通过定时任务监控可用数量。

package nexus.io.playwright.pool;

import java.util.Random;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;

import com.microsoft.playwright.Browser;
import com.microsoft.playwright.BrowserContext;
import com.microsoft.playwright.BrowserType;
import com.microsoft.playwright.Page;
import com.microsoft.playwright.Playwright;

import lombok.extern.slf4j.Slf4j;

import com.microsoft.playwright.BrowserType.LaunchOptions;

/**
 * PlaywrightPool 用于管理 BrowserContext 对象，减少频繁创建造成的性能开销。
 * 每次 acquirePage() 从池中取出一个 BrowserContext，并创建一个 Page，
 * 当 Page 关闭时自动将 BrowserContext 归还池中。
 */
@Slf4j
public class PlaywrightPool {
  private static BlockingQueue<BrowserContext> pool = null;
  private static BlockingQueue<Playwright> playwrightPool = null;
  private static BlockingQueue<Browser> browserPool = null;
  private static int poolSize = 0;
  public static LaunchOptions launchOptions = new BrowserType.LaunchOptions().setHeadless(true);

  private static ScheduledExecutorService scheduler = Executors.newSingleThreadScheduledExecutor(Thread.ofVirtual().factory());

  /**
   * 构造时初始化 Playwright、Browser 以及固定数量的 BrowserContext
   *
   * @param poolSize 池大小
   */
  public static void init(int poolSize) {
    PlaywrightPool.poolSize = poolSize;

    PlaywrightPool.pool = new ArrayBlockingQueue<>(poolSize);
    PlaywrightPool.playwrightPool = new ArrayBlockingQueue<>(poolSize);
    PlaywrightPool.browserPool = new ArrayBlockingQueue<>(poolSize);
    for (int i = 0; i < poolSize; i++) {
      Playwright playwright = Playwright.create();
      Browser browser = playwright.chromium().launch(launchOptions);
      BrowserContext context = browser.newContext();
      playwrightPool.offer(playwright);
      browserPool.offer(browser);
      pool.offer(context);
    }

    scheduler.scheduleAtFixedRate(() -> {
      new Random().nextInt(1,10);
      log.info("PlaywrightPool - Available: {}/{}", PlaywrightPool.availableCount(), PlaywrightPool.totalCount());
    }, 0, 30, TimeUnit.SECONDS);
  }

  /**
   * 获取一个 Page 对象，内部会从池中取出一个 BrowserContext，
   * 并包装为 PooledPage（实现了 Page 接口、AutoCloseable）。
   *
   * @return PooledPage 对象，使用完毕后调用 close() 归还 BrowserContext
   * @throws InterruptedException 如果等待过程中被中断
   */
  public static Page acquirePage() throws InterruptedException {
    BrowserContext context = pool.take();
    Page page = context.newPage();
    return new PooledPage(page, context, pool);
  }

  /**
   * 返回当前池中可用的 BrowserContext 数量
   *
   * @return 可用数量
   */
  public static int availableCount() {
    return pool.size();
  }

  /**
   * 池的总大小
   *
   * @return 池大小
   */
  public static int totalCount() {
    return poolSize;
  }

  /**
   * 关闭池中所有 BrowserContext 以及 Browser、Playwright 实例
   */
  public static void close() {
    for (BrowserContext context : pool) {
      context.close();
    }

    for (Playwright context : playwrightPool) {
      context.close();
    }

    for (Browser context : browserPool) {
      context.close();
    }
  }

}

3.2 PooledPage

包装了 Playwright 的 Page 对象，当 Page 关闭时自动归还 BrowserContext 到池中，从而简化资源管理。代码太长,省略

4. 数据持久化与 DAO

WebPageDao 类用于将页面数据（HTML、Markdown 等）写入数据库，同时利用 Striped 锁保证同一 URL 的并发写入安全。

package nexus.io.playwright.dao;

import java.util.concurrent.locks.Lock;

import org.jsoup.Jsoup;

import com.google.common.util.concurrent.Striped;
import nexus.io.db.activerecord.Db;
import nexus.io.db.activerecord.Row;
import nexus.io.playwright.utils.WebsiteUrlUtils;
import nexus.io.tio.utils.snowflake.SnowflakeIdUtils;

public class WebPageDao {
  private final Striped<Lock> stripedLocks = Striped.lock(256);

  public boolean exists(String cacheTableName, String field, String value) {
    return Db.exists(cacheTableName, field, value);
  }

  public void saveMarkdown(String tableName, String url, String title, String type, String markdown) {
    String canonical = WebsiteUrlUtils.canonicalizeUrl(url);
    if (exists(tableName, "url", canonical)) {
      return;
    }

    Lock lock = stripedLocks.get(canonical);
    lock.lock();
    try {
      if (exists(tableName, "url", canonical)) {
        return;
      }
      Row row = new Row().set("id", SnowflakeIdUtils.id()).set("url", canonical).set("title", title).set("type", type).set("markdown", markdown);
      Db.save(tableName, row);
    } finally {
      lock.unlock();
    }
  }

  /**
   * 保存页面内容到数据库（使用 Striped 锁防止并发写入同一 URL）
   */
  public void saveContent(String tableName, String url, String title, String type, String html) {

    String canonical = WebsiteUrlUtils.canonicalizeUrl(url);
    if (exists(tableName, "url", canonical)) {
      return;
    }

    Lock lock = stripedLocks.get(canonical);
    lock.lock();
    try {
      if (exists(tableName, "url", canonical)) {
        return;
      }
      Row row = new Row().set("id", SnowflakeIdUtils.id()).set("url", canonical).set("title", title).set("type", type).set("html", html).set("text", Jsoup.parse(html).text());
      Db.save(tableName, row);
    } finally {
      lock.unlock();
    }
  }

  public void saveHtmlAndMarkdown(String tableName, String url, String title, String html, String markdown) {
    String canonical = WebsiteUrlUtils.canonicalizeUrl(url);
    if (exists(tableName, "url", canonical)) {
      return;
    }

    Lock lock = stripedLocks.get(canonical);
    lock.lock();
    try {
      if (exists(tableName, "url", canonical)) {
        return;
      }
      Row row = new Row().set("id", SnowflakeIdUtils.id()).set("url", canonical).set("title", title).set("type", "html")
          //
          .set("html", html).set("markdown", markdown);
      Db.save(tableName, row);
    } finally {
      lock.unlock();
    }
  }

}

5. 爬虫服务

5.1 WebPageCrawlService

负责爬取页面数据，包括针对 PDF 与 HTML 页面分别处理。对于已存在的 URL，先从数据库缓存中直接获取数据；否则利用浏览器池打开页面进行爬取、等待页面加载完成后提取 HTML 和标题，再保存至数据库。

package nexus.io.playwright.service;

import java.util.concurrent.locks.Lock;

import com.google.common.util.concurrent.Striped;
import nexus.io.db.activerecord.Db;
import nexus.io.db.activerecord.Row;
import nexus.io.model.web.WebPageContent;
import nexus.io.playwright.consts.TableNames;
import nexus.io.playwright.pool.PlaywrightPool;
import nexus.io.playwright.utils.PDFUtils;
import nexus.io.tio.utils.hutool.FilenameUtils;
import nexus.io.tio.utils.snowflake.SnowflakeIdUtils;
import com.microsoft.playwright.Page;
import com.microsoft.playwright.options.LoadState;

public class WebPageCrawlService {

  private final Striped<Lock> stripedLocks = Striped.lock(256);

  /**
   * cache
   * @param url
   * @return
   */
  public String getPdfContent(String url) {
    Lock lock = stripedLocks.get(url);
    lock.lock();
    try {
      String sql = "select html from %s where type=? and url=?";
      sql = String.format(sql, TableNames.web_page_cache);
      String title = FilenameUtils.getBaseName(url);
      //db cache
      String content = Db.queryStr(sql, "pdf", url);
      if (content != null) {
        return content;
      }
      // http
      content = PDFUtils.getContent(url);
      content = content.replace("\u0000", "").trim();
      Row row = new Row().set("id", SnowflakeIdUtils.id()).set("url", url).set("title", title).set("type", "pdf")
          //
          .set("html", content);
      Db.save(TableNames.web_page_cache, row);
      return content;
    } finally {
      lock.unlock();
    }

  }

  public WebPageContent getHtml(String url) throws InterruptedException {
    Lock lock = stripedLocks.get(url);
    lock.lock();
    try {
      String sql = "select title,html from %s where type=? and url=?";
      sql = String.format(sql, TableNames.web_page_cache);
      //db cache
      Row first = Db.findFirst(sql, "html", url);

      if (first != null) {
        String title = first.getStr("title");
        String html = first.getString("html");
        WebPageContent content = new WebPageContent(title, url).setContent(html);
        return content;
      }
      // http
      try (Page page = PlaywrightPool.acquirePage()) {
        // 设置页面超时时间为 1 分钟（60000ms）
        page.setDefaultNavigationTimeout(60000);
        page.setDefaultTimeout(60000);
        // 控制爬取速率
        try {
          Thread.sleep(500);
        } catch (InterruptedException e) {
          e.printStackTrace();
        }
        // 导航至目标 URL
        page.navigate(encodeUrl(url));
        // 等待页面达到网络空闲状态和加载完成状态
        page.waitForLoadState(LoadState.NETWORKIDLE);
        page.waitForLoadState(LoadState.LOAD);
        String html = page.content();
        String title = page.title();

        Row row = new Row().set("id", SnowflakeIdUtils.id()).set("url", url).set("title", title).set("type", "html")
            //
            .set("html", html);
        Db.save(TableNames.web_page_cache, row);
        return new WebPageContent(title, url, "", html);
      }
    } finally {
      lock.unlock();
    }
  }

  /**
   * 对 URL 进行预处理，针对路径中的非法字符进行编码。
   *
   * @param url 原始 URL
   * @return 处理后的 URL
   */
  private String encodeUrl(String url) {
    if (url == null || url.isEmpty()) {
      return url;
    }
    return url.replace("[", "%5B").replace("]", "%5D");
  }
}

5.2 CrawlWebPageTask

此类为爬虫任务的核心逻辑，负责从数据库任务队列中取出待爬取 URL，然后根据页面类型（HTML 或 PDF）调用对应的处理方法。
在处理 HTML 页面时，还会利用 Jsoup 提取页面中所有有效链接（只保留与目标域名相同的链接）并加入任务队列中，从而保证整个网站的完整爬取。

package nexus.io.playwright.service;

import java.util.HashSet;
import java.util.List;
import java.util.Set;

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;

import nexus.io.db.activerecord.Db;
import nexus.io.jfinal.aop.Aop;
import nexus.io.model.web.WebPageContent;
import nexus.io.playwright.dao.WebPageDao;
import nexus.io.playwright.model.WebPageUrl;
import nexus.io.playwright.utils.MarkdownUtils;
import nexus.io.playwright.utils.TaskExecutorUtils;
import nexus.io.playwright.utils.WebsiteUrlUtils;
import nexus.io.tio.utils.hutool.FilenameUtils;
import nexus.io.tio.utils.hutool.StrUtil;
import nexus.io.tio.utils.snowflake.SnowflakeIdUtils;

import lombok.extern.slf4j.Slf4j;

@Slf4j
public class CrawlWebPageTask {

  WebPageCrawlService crawlWebPageService = Aop.get(WebPageCrawlService.class);
  WebPageDao webPageDao = Aop.get(WebPageDao.class);
  String tableName;
  String baseUrl;
  String baseDomain;

  public CrawlWebPageTask(String url, String tableName) {
    this.baseUrl = url;
    this.tableName = tableName;
    String domain = WebsiteUrlUtils.extractDomain(url);
    this.baseDomain = domain;
    this.addLink(url);
  }

  public void run() {

    while (true) {
      /**
       * status 0 add 1 starting 2.finish
       */
      String sql = "select id,url,status,tried from web_page_url where status=0 and tried < 3 limit 10";
      List<WebPageUrl> list = WebPageUrl.dao.find(sql);
      if (list.size() > 0) {
        for (WebPageUrl webPageUrl : list) {
          this.updateUrlStatusToRunning(webPageUrl);
          TaskExecutorUtils.executor.submit(() -> {
            try {
              this.processUrl(webPageUrl);
            } catch (Exception e) {
              log.error(e.getMessage(), e);
            }
          });
        }
      }
      try {
        Thread.sleep(1000L);
      } catch (InterruptedException e) {
        e.printStackTrace();
      }
    }
  }

  public void processUrl(WebPageUrl webPageUrl) {
    String url = "https://" + webPageUrl.getUrl();
    // 针对 PDF 与 HTML 分支分别处理
    if (url.endsWith(".pdf")) {
      try {
        // 处理 PDF 文件
        String content = Aop.get(WebPageCrawlService.class).getPdfContent(url);
        String filename = FilenameUtils.getBaseName(url);
        if (content != null) {
          content = content.replace("\u0000", "").trim();
          try {
            webPageDao.saveMarkdown(tableName, url, filename, "pdf", content);
            updateUrlStatusToFinished(webPageUrl);
          } catch (Exception e) {
            log.error("Failed to save:{},{}", url, filename, e);
          }

        }
      } catch (Exception e) {
        log.error("PDF processing failed for URL {}: {}", url, e.getMessage());
        updateFailureCount(webPageUrl);
      }
    } else {
      try {
        String canonical = WebsiteUrlUtils.canonicalizeUrl(url);
        if (webPageDao.exists(tableName, "url", canonical)) {
          return;
        }
        log.info("Processing URL: {} (Attempt {})", url, webPageUrl.getTried());
        WebPageContent webPage = Aop.get(WebPageCrawlService.class).getHtml(url);
        String title = webPage.getTitle();
        String html = webPage.getContent();
        // 假设 htmlString 是的 HTML 字符串
        Document document = Jsoup.parse(html, baseUrl);
        Element body = document.body();
        String bodyHtml = body.html();
        String markdown = MarkdownUtils.toMd(bodyHtml);
        try {
          webPageDao.saveHtmlAndMarkdown(tableName, url, title, html, markdown);
          updateUrlStatusToFinished(webPageUrl);
        } catch (Exception e) {
          log.error("Failed to save:{},{}", url, title, e);
        }
        Set<String> links = extractValidLinks(document);
        for (String link : links) {
          addLink(link);
        }
      } catch (Exception e) {
        log.error(e.getMessage(), e);
        updateFailureCount(webPageUrl);
      }
    }
  }

  public void addLink(String link) {
    String canonical = WebsiteUrlUtils.canonicalizeUrl(link);
    boolean urlExists = webPageDao.exists("web_page_url", "url", canonical);
    if (!urlExists) {
      new WebPageUrl().setId(SnowflakeIdUtils.id()).setUrl(canonical).save();
    }
  }

  /**
   * 使用 Jsoup 解析 DOM，提取页面中所有有效链接（剔除锚点、只保留同一域链接）
   */
  public Set<String> extractValidLinks(Document doc) {
    Set<String> links = new HashSet<>();
    Elements elements = doc.select("a[href]");
    for (Element el : elements) {
      String absUrl = el.absUrl("href");
      String normalized = WebsiteUrlUtils.normalizeUrl(absUrl);
      if (StrUtil.isBlank(normalized)) {
        continue;
      }
      if (isSameDomain(normalized)) {
        links.add(normalized);
      }
    }
    return links;
  }

  /**
   * 判断 URL 是否与基础域名相同
   */
  private boolean isSameDomain(String url) {
    String domain = WebsiteUrlUtils.extractDomain(url);
    return domain.equalsIgnoreCase(baseDomain);
  }

  private void updateUrlStatusToRunning(WebPageUrl webPageUrl) {
    Db.updateBySql("update web_page_url set tried=1,status=1 where id=?", webPageUrl.getId());
  }

  private void updateUrlStatusToFinished(WebPageUrl webPageUrl) {
    Db.updateBySql("update web_page_url set tried=1,status=2 where id=?", webPageUrl.getId());
  }

  private void updateFailureCount(WebPageUrl webPageUrl) {
    Integer tried = webPageUrl.getTried();
    tried++;
    Db.updateBySql("update web_page_url set status =0,tried=? where id=?", tried, webPageUrl.getId());
  }

}

6. 控制器入口

通过 CrawlController 提供一个 HTTP GET 接口，当请求此接口时，会启动爬虫任务，开始对目标网站进行爬取。

package nexus.io.playwright.controller;

import nexus.io.annotation.Get;
import nexus.io.annotation.RequestPath;
import nexus.io.model.body.RespBodyVo;
import nexus.io.playwright.consts.TableNames;
import nexus.io.playwright.service.CrawlWebPageTask;
import nexus.io.tio.utils.thread.TioThreadUtils;

import lombok.extern.slf4j.Slf4j;

@RequestPath("/crawl")
@Slf4j
public class CrawlController {

  @Get("/hawaii_kapiolani_web_page")
  public RespBodyVo index() {
    TioThreadUtils.execute(() -> {
      String url = "https://www.kapiolani.hawaii.edu/";
      // AdvancedCrawlService 构造时启动爬虫任务
      CrawlWebPageTask crawlWebPageTask = new CrawlWebPageTask(url, TableNames.hawaii_kapioalni_web_page);
      try {
        crawlWebPageTask.run();
      } catch (Exception e) {
        log.error(e.getMessage(), e);
      }
    });
    return RespBodyVo.ok();
  }
}

7. 工具类与辅助功能

7.1 MarkdownUtils

利用 Flexmark 将 HTML 内容转换为 Markdown 格式。

package nexus.io.playwright.utils;

import com.vladsch.flexmark.html2md.converter.FlexmarkHtmlConverter;

public class MarkdownUtils {
  public static FlexmarkHtmlConverter converter = FlexmarkHtmlConverter.builder().build();

  public static String toMd(String html) {
    return converter.convert(html);
  }
}

7.2 PDFUtils

使用 PDFBox 解析 PDF 文件，并提取文本内容。

package nexus.io.playwright.utils;

import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;

import org.apache.pdfbox.pdmodel.PDDocument;
import org.apache.pdfbox.text.PDFTextStripper;

import nexus.io.tio.utils.http.HttpDownloadUtils;
import nexus.io.tio.utils.url.UrlUtils;

public class PDFUtils {
  public static String getContent(String rawUrl) {
    try {
      String url = UrlUtils.encodeUrl(rawUrl);
      ByteArrayOutputStream download = HttpDownloadUtils.download(url, null);
      try (ByteArrayInputStream inputStream = new ByteArrayInputStream(download.toByteArray()); PDDocument document = PDDocument.load(inputStream)) {
        PDFTextStripper pdfStripper = new PDFTextStripper();
        String text = pdfStripper.getText(document);
        return text;
      }
    } catch (IOException e) {
      throw new RuntimeException(e);
    }
  }
}

7.3 TaskExecutorUtils

管理线程池，利用 Java 虚拟线程执行爬虫任务，同时设置队列容量和拒绝执行策略。

package nexus.io.playwright.utils;

import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.RejectedExecutionException;
import java.util.concurrent.RejectedExecutionHandler;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;

public class TaskExecutorUtils {
  public static int cpuCount = Runtime.getRuntime().availableProcessors();
  private static AtomicLong threadCounter = new AtomicLong(0);

  private static int queueCapacity = 100;

  public static ExecutorService executor;

  static {
    executor = new ThreadPoolExecutor(cpuCount, // corePoolSize
        cpuCount, // maximumPoolSize
        0L, // keepAliveTime
        TimeUnit.MILLISECONDS, // time unit
        new ArrayBlockingQueue<>(queueCapacity), //
        runnable -> {
          Thread t = Thread.ofVirtual().factory().newThread(runnable);
          t.setName("crawl-thread-" + threadCounter.getAndIncrement());
          return t;
        },

        new RejectedExecutionHandler() {
          @Override
          public void rejectedExecution(Runnable r, ThreadPoolExecutor executor) {
            try {
              executor.getQueue().put(r);
            } catch (InterruptedException e) {
              Thread.currentThread().interrupt();
              throw new RejectedExecutionException("Task submission interrupted", e);
            }
          }
        });
  }
}

7.4 WebsiteUrlUtils

提供 URL 的域名提取、标准化和规范化方法，确保相同页面不会重复爬取。

package nexus.io.playwright.utils;

import java.net.URL;

import lombok.extern.slf4j.Slf4j;

@Slf4j
public class WebsiteUrlUtils {

  /**
   * 提取 URL 中的域名（不含 www. 前缀）
   */
  public static String extractDomain(String url) {
    try {
      URL netUrl = new URL(url);
      String host = netUrl.getHost();
      if (host != null) {
        return host.startsWith("www.") ? host.substring(4) : host;
      }
    } catch (Exception e) {
      log.error("Error extracting domain from url: {}", url, e);
    }
    return "";
  }

  /**
   * 生成 URL 的标准形式，去除协议部分（http, https）
   */
  public static String canonicalizeUrl(String url) {
    String normalized = normalizeUrl(url);
    return normalized.replaceFirst("(?i)^(https?://)", "");
  }

  /**
   * 规范化 URL：剔除锚点部分，去除尾部斜杠，并做 trim
   */
  public static String normalizeUrl(String url) {
    if (url == null || url.isEmpty())
      return "";
    int index = url.indexOf("#");
    if (index != -1) {
      url = url.substring(0, index);
    }
    url = url.trim();
    if (url.endsWith("/") && url.length() > 1) {
      url = url.substring(0, url.length() - 1);
    }
    return url;
  }
}

8. 内部任务模型

CrawlTask 类记录了待爬取 URL 及其爬取深度，便于后续扩展任务调度逻辑。

package nexus.io.playwright.vo;

import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;

/**
 * 内部任务类，记录 URL 及其爬取深度
 */
@Data
@NoArgsConstructor
@AllArgsConstructor
public class CrawlTask {
  private String url;
  private int depth;
}

总结

本文档详细介绍了如何利用 Playwright 结合 Java 21 虚拟线程来实现一个静态网站数据爬取系统。我们提供了数据库表结构、系统配置、浏览器上下文池管理、任务调度、页面数据获取、数据持久化以及辅助工具类的完整代码，并对各模块的作用和实现逻辑进行了详细说明。整个系统既考虑到了爬取的高并发性，也注重了数据去重、异常处理和后续数据转换，为大模型应用提供了坚实的数据支持。