Adds resource-level memoization for full crawl runs so unchanged employee profiles can be skipped without creating redundant snapshots.
What changed
Added parse_resource_cache storage for crawled profile resources.
Added conditional HTTP requests with ETag / Last-Modified support.
Reuses cached response bodies on 304 Not Modified.
Tracks profile resources used during parsing: main HTML, publications API, and graduation theses API.
Skips new EmployeeSnapshot creation when parsed checksum and parser version are unchanged.
Added crawl_runs.skipped_count and surfaced it in admin/API/MCP progress payloads.
Normalized date-dependent experience text for checksums, so automatic “years of experience” drift does not count as profile changes.
Updated README MCP tools list and parsing progress docs.
Bumped service version to 0.6.0.
Testing
Added coverage for conditional cache reuse via ETag.
Added coverage for skipping snapshots when checksum is unchanged.
Added checksum regression tests for widget changes and date-dependent experience text.
Updated progress/version tests.
Validation run:
.venv\Scripts\python.exe -m pytest
## Summary
Adds resource-level memoization for full crawl runs so unchanged employee profiles can be skipped without creating redundant snapshots.
## What changed
- Added `parse_resource_cache` storage for crawled profile resources.
- Added conditional HTTP requests with `ETag` / `Last-Modified` support.
- Reuses cached response bodies on `304 Not Modified`.
- Tracks profile resources used during parsing: main HTML, publications API, and graduation theses API.
- Skips new `EmployeeSnapshot` creation when parsed checksum and parser version are unchanged.
- Added `crawl_runs.skipped_count` and surfaced it in admin/API/MCP progress payloads.
- Normalized date-dependent experience text for checksums, so automatic “years of experience” drift does not count as profile changes.
- Updated README MCP tools list and parsing progress docs.
- Bumped service version to `0.6.0`.
## Testing
- Added coverage for conditional cache reuse via `ETag`.
- Added coverage for skipping snapshots when checksum is unchanged.
- Added checksum regression tests for widget changes and date-dependent experience text.
- Updated progress/version tests.
Validation run:
```bash
.venv\Scripts\python.exe -m pytest
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Summary
Adds resource-level memoization for full crawl runs so unchanged employee profiles can be skipped without creating redundant snapshots.
What changed
parse_resource_cachestorage for crawled profile resources.ETag/Last-Modifiedsupport.304 Not Modified.EmployeeSnapshotcreation when parsed checksum and parser version are unchanged.crawl_runs.skipped_countand surfaced it in admin/API/MCP progress payloads.0.6.0.Testing
ETag.Validation run: