feat: add detailed employee publications storage and MCP docs

Reviewed-on: #26

.env.example

@@ -14,13 +14,5 @@ PARSER_USE_PLAYWRIGHT=false
 ADMIN_USERNAME=admin
 ADMIN_PASSWORD=change-me
 SESSION_SECRET=change-me-session-secret
-MCP_TOKEN=change-me-mcp-token
-MCP_AUTH_MODE=oauth
-MCP_RESOURCE_URL=http://localhost:8001/mcp
-MCP_OAUTH_ISSUER=
-MCP_OAUTH_AUDIENCE=
-MCP_OAUTH_JWKS_URL=
-MCP_OAUTH_REQUIRED_SCOPE=mcp:tools
-
 API_PORT=8000
 MCP_PORT=8001

.gitignore

@@ -8,3 +8,4 @@ pytest-cache-files-*/
 .coverage
 htmlcov/
 postgres_data/
+MCP_DESCRIPTION.md

MCP_DESCRIPTION.md

@@ -0,0 +1,648 @@
+# MCP: описание работы, структуры и тулзов
+
+Документ описывает MCP endpoint сервиса `miem-employees` по текущей реализации в `app/mcp.py`.
+
+## Где находится MCP
+
+- FastAPI router: `app.mcp.router`
+- Подключение к приложению: `app/main.py`
+- HTTP endpoint: `POST /mcp`
+- Локально при обычном запуске API: `http://localhost:8000/mcp`
+- В Docker Compose через отдельный сервис `mcp`: `http://localhost:8001/mcp`
+- Авторизация на уровне приложения: отсутствует. Заголовок `Authorization` не проверяется и не влияет на ответ.
+
+Если доступ к MCP нужно ограничить, это должно делаться внешним контуром: bind на localhost, VPN, firewall, reverse proxy или отдельная сетевая политика.
+
+## Протокол
+
+Endpoint принимает JSON-RPC 2.0 over HTTP.
+
+Общий формат запроса:
+
+```json
+{
+  "jsonrpc": "2.0",
+  "id": 1,
+  "method": "tools/list",
+  "params": {}
+}
+```
+
+Общий формат успешного ответа:
+
+```json
+{
+  "jsonrpc": "2.0",
+  "id": 1,
+  "result": {}
+}
+```
+
+Общий формат ошибки:
+
+```json
+{
+  "jsonrpc": "2.0",
+  "id": 1,
+  "error": {
+    "code": -32601,
+    "message": "Method not found"
+  }
+}
+```
+
+Поддерживаемая версия MCP-протокола:
+
+```text
+2024-11-05
+```
+
+Имя сервиса:
+
+```text
+miem-employees
+```
+
+Версия сервера берется из `app.version.BACKEND_VERSION`.
+
+## Поддерживаемые JSON-RPC методы
+
+### initialize
+
+Возвращает метаданные MCP-сервера и capabilities.
+
+Запрос:
+
+```json
+{
+  "jsonrpc": "2.0",
+  "id": 1,
+  "method": "initialize",
+  "params": {}
+}
+```
+
+Ответ:
+
+```json
+{
+  "jsonrpc": "2.0",
+  "id": 1,
+  "result": {
+    "protocolVersion": "2024-11-05",
+    "serverInfo": {
+      "name": "miem-employees",
+      "version": "0.5.0"
+    },
+    "capabilities": {
+      "tools": {}
+    }
+  }
+}
+```
+
+### tools/list
+
+Возвращает список доступных tools с JSON Schema для аргументов.
+
+Запрос:
+
+```json
+{
+  "jsonrpc": "2.0",
+  "id": 1,
+  "method": "tools/list",
+  "params": {}
+}
+```
+
+Ответ содержит массив `result.tools`.
+
+### tools/call
+
+Вызывает один tool по имени.
+
+Запрос:
+
+```json
+{
+  "jsonrpc": "2.0",
+  "id": 1,
+  "method": "tools/call",
+  "params": {
+    "name": "search_employees",
+    "arguments": {
+      "query": "Сергеев",
+      "limit": 20
+    }
+  }
+}
+```
+
+Ответ tool всегда заворачивается в MCP content-массив:
+
+```json
+{
+  "jsonrpc": "2.0",
+  "id": 1,
+  "result": {
+    "content": [
+      {
+        "type": "text",
+        "text": "{\"items\":[]}"
+      }
+    ]
+  }
+}
+```
+
+Поле `text` содержит сериализованный JSON с `ensure_ascii=false`. Клиент должен распарсить это поле как JSON, если ему нужна структурированная нагрузка.
+
+## Ошибки
+
+- Неизвестный JSON-RPC метод: `code = -32601`, `message = "Method not found"`.
+- Исключения при обработке tool: `code = -32000`, `message` содержит текст исключения.
+- Если сущность не найдена внутри отдельных tools, HTTP и JSON-RPC ответ остаются успешными, а полезная нагрузка содержит `{"error": "not_found"}`.
+
+## Источники данных
+
+MCP читает данные из основной базы через SQLAlchemy session из `app.db.get_db`.
+
+Основные таблицы и модели:
+
+- `employees`: текущая карточка сотрудника, статус, профиль, `current_data`, checksum.
+- `employee_publications`: нормализованные публикации сотрудников с авторами, DOI, аннотацией, описанием, citation text и raw JSON из HSE Publications.
+- `crawl_runs`: история запусков парсинга.
+- `crawl_run_employee_changes`: детальные изменения сотрудников в рамках запуска.
+- `crawl_errors`: ошибки парсинга в рамках запуска.
+- `dataset_versions`: версии полного набора сотрудников.
+- `dataset_version_items`: состав конкретной версии набора сотрудников.
+
+## Общая структура employee payload
+
+Краткая карточка сотрудника:
+
+```json
+{
+  "profile_key": "staff:avsergeev",
+  "profile_id": "avsergeev",
+  "full_name": "Сергеев Алексей Викторович",
+  "status": "active",
+  "canonical_url": "https://www.hse.ru/staff/avsergeev",
+  "last_seen_at": "2026-05-14T10:00:00+00:00",
+  "dismissed_at": null
+}
+```
+
+В sync payload дополнительно отдается `checksum`.
+
+Полная карточка дополнительно содержит:
+
+```json
+{
+  "data": {
+    "contacts": {},
+    "sections": []
+  }
+}
+```
+
+`data` соответствует распарсенному JSON профиля сотрудника. Внутри `sections` могут быть секции с публикациями, курсами, ВКР, таблицами, ссылками и произвольными текстовыми блоками.
+
+## Tools
+
+### get_service_info
+
+Назначение: вернуть метаданные сервиса, список tools и текущую версию набора сотрудников.
+
+Аргументы: отсутствуют.
+
+Возвращает:
+
+```json
+{
+  "service_name": "miem-employees",
+  "backend_version": "0.5.0",
+  "protocolVersion": "2024-11-05",
+  "tools": [],
+  "dataset": {
+    "hash": "sha256",
+    "previous_hash": "sha256 или null",
+    "created_at": "2026-05-14T10:00:00+00:00",
+    "crawl_run_id": 123,
+    "employee_count": 100,
+    "active_count": 95,
+    "dismissed_count": 5
+  }
+}
+```
+
+Особенность: перед ответом сервис создает актуальную `dataset_version`, если текущий набор сотрудников еще не имеет версии.
+
+### sync_employees
+
+Назначение: синхронизировать клиентский кэш сотрудников по hash набора данных.
+
+Аргументы:
+
+```json
+{
+  "client_hash": "sha256 или null",
+  "include_data": true
+}
+```
+
+- `client_hash`: hash версии, которая уже есть у клиента. Если не передан, отдается полный snapshot.
+- `include_data`: управляет включением полного `data` в карточки сотрудников. По умолчанию `true`.
+
+Полный ответ без `client_hash`:
+
+```json
+{
+  "mode": "full",
+  "from_hash": null,
+  "to_hash": "current-sha256",
+  "dataset": {},
+  "items": []
+}
+```
+
+Если клиентский hash совпадает с текущим:
+
+```json
+{
+  "mode": "delta",
+  "from_hash": "current-sha256",
+  "to_hash": "current-sha256",
+  "dataset": {},
+  "changes": {
+    "added": [],
+    "updated": [],
+    "dismissed": [],
+    "removed": []
+  }
+}
+```
+
+Если `client_hash` неизвестен серверу:
+
+```json
+{
+  "mode": "full",
+  "from_hash": "missing",
+  "to_hash": "current-sha256",
+  "dataset": {},
+  "items": [],
+  "reason": "unknown_client_hash"
+}
+```
+
+Если `client_hash` найден и отличается от текущего:
+
+```json
+{
+  "mode": "delta",
+  "from_hash": "old-sha256",
+  "to_hash": "current-sha256",
+  "dataset": {},
+  "changes": {
+    "added": [],
+    "updated": [],
+    "dismissed": [],
+    "removed": []
+  }
+}
+```
+
+Логика delta:
+
+- `added`: сотрудник появился в новой версии.
+- `updated`: изменился checksum или статус, и сотрудник активен.
+- `dismissed`: сотрудник есть в новой версии, но получил статус `dismissed`.
+- `removed`: `profile_key` был в старой версии, но отсутствует в новой.
+
+Hash набора считается по отсортированному списку `{profile_key, status, checksum}`.
+
+### search_employees
+
+Назначение: найти сотрудников по ФИО или canonical URL.
+
+Аргументы:
+
+```json
+{
+  "query": "Сергеев",
+  "status": "active",
+  "limit": 20
+}
+```
+
+- `query`: обязательный по schema, но в коде пустая строка означает поиск без текстового фильтра.
+- `status`: опционально, только `active` или `dismissed`.
+- `limit`: максимум 100, по умолчанию 20.
+
+Возвращает массив кратких employee payload без `data`:
+
+```json
+[
+  {
+    "profile_key": "staff:avsergeev",
+    "profile_id": "avsergeev",
+    "full_name": "Сергеев Алексей Викторович",
+    "status": "active",
+    "canonical_url": "https://www.hse.ru/staff/avsergeev",
+    "last_seen_at": "2026-05-14T10:00:00+00:00",
+    "dismissed_at": null
+  }
+]
+```
+
+### get_employee
+
+Назначение: получить одну карточку сотрудника.
+
+Аргументы:
+
+```json
+{
+  "profile_id_or_url": "avsergeev"
+}
+```
+
+Поиск выполняется по:
+
+- `profile_key`
+- `profile_id`
+- точному `canonical_url`
+- частичному совпадению `canonical_url`
+
+Возвращает полный employee payload с `data`.
+
+Если сотрудник не найден:
+
+```json
+{
+  "error": "not_found"
+}
+```
+
+### list_employee_publications
+
+Назначение: вернуть публикации сотрудника. Если есть нормализованные строки в `employee_publications`, tool возвращает детальные публикационные данные: авторов, DOI, аннотацию, описание, citation text, год, тип, язык, статус и ссылки. Если детальная таблица еще не заполнена, tool использует старый fallback из `employees.current_data.sections[].publications`.
+
+Аргументы:
+
+```json
+{
+  "profile_id_or_url": "avsergeev"
+}
+```
+
+Поиск сотрудника выполняется так же, как в `get_employee`: по `profile_key`, `profile_id`, точному или частичному `canonical_url`.
+
+Порядок источников:
+
+- сначала `employee_publications`, отсортированные по году, названию и внутреннему id;
+- если записей нет, секции `current_data.sections` с `type = "publications"` и массивами `publications`.
+
+Ответ:
+
+```json
+{
+  "employee": {
+    "profile_key": "org_person:803294906",
+    "profile_id": "803294906",
+    "full_name": "Борисов Сергей Петрович",
+    "status": "active",
+    "canonical_url": "https://www.hse.ru/org/persons/803294906",
+    "last_seen_at": "2026-05-14T10:00:00+00:00",
+    "dismissed_at": null
+  },
+  "items": [
+    {
+      "id": "888959076",
+      "publication_id": "888959076",
+      "title": "Название публикации",
+      "text": "Краткое описание или citation",
+      "url": "https://publications.hse.ru/view/888959076",
+      "year": 2023,
+      "type": "ARTICLE",
+      "publication_type": "ARTICLE",
+      "language": "ru",
+      "status": 1,
+      "doi_url": "https://doi.org/10.53921/18195822_2023_23_4_624",
+      "other_url": "https://example.test",
+      "document_url": "https://example.test/file.pdf",
+      "citation_text": "Авторы. Название публикации // Журнал. 2023.",
+      "annotation": {
+        "ru": "Аннотация",
+        "en": "Abstract"
+      },
+      "description": {
+        "main": "Авторы. Название публикации // Журнал. 2023."
+      },
+      "authors": [
+        {
+          "id": "803294906",
+          "href": "https://www.hse.ru/org/persons/803294906",
+          "title_ru": "Борисов С. П.",
+          "title_en": "",
+          "reverse_title_ru": "С. П. Борисов",
+          "reverse_title_en": "",
+          "alt_name": "S. P. Borisov",
+          "other_name": null,
+          "is_current_employee": true
+        }
+      ]
+    }
+  ]
+}
+```
+
+В fallback-режиме из `current_data` старые элементы могут содержать только базовые поля `title`, `text`, `url` и `id`.
+
+Если сотрудник не найден:
+
+```json
+{
+  "items": []
+}
+```
+
+Если сотрудник найден, но публикаций нет:
+
+```json
+{
+  "employee": {},
+  "items": []
+}
+```
+
+### list_employee_courses
+
+Назначение: вернуть курсы преподавания сотрудника из распарсенных секций профиля.
+
+Аргументы:
+
+```json
+{
+  "profile_id_or_url": "avsergeev"
+}
+```
+
+Сервис ищет секции `current_data.sections` с `type = "courses_by_year"` и объединяет массивы `courses`.
+
+Ответ:
+
+```json
+{
+  "employee": {},
+  "items": [
+    {
+      "title": "Название курса",
+      "url": "https://..."
+    }
+  ]
+}
+```
+
+Если сотрудник или данные профиля отсутствуют:
+
+```json
+{
+  "items": []
+}
+```
+
+### get_crawl_status
+
+Назначение: вернуть последний запуск парсинга.
+
+Аргументы: отсутствуют.
+
+Ответ:
+
+```json
+{
+  "id": 123,
+  "status": "completed",
+  "source_url": "https://miem.hse.ru/persons",
+  "started_at": "2026-05-14T10:00:00+00:00",
+  "finished_at": "2026-05-14T10:10:00+00:00",
+  "found_count": 100,
+  "parsed_count": 98,
+  "error_count": 2,
+  "dismissed_count": 1
+}
+```
+
+Если запусков еще не было:
+
+```json
+{
+  "status": "never_run"
+}
+```
+
+### get_crawl_run_details
+
+Назначение: вернуть детальную информацию по конкретному запуску парсинга: summary, изменения сотрудников и ошибки.
+
+Аргументы:
+
+```json
+{
+  "run_id": 123
+}
+```
+
+Ответ:
+
+```json
+{
+  "id": 123,
+  "source_url": "https://miem.hse.ru/persons",
+  "status": "completed",
+  "status_display": "Завершен",
+  "started_at": "2026-05-14T10:00:00+00:00",
+  "finished_at": "2026-05-14T10:10:00+00:00",
+  "started_display": "14.05.2026 13:00",
+  "finished_display": "14.05.2026 13:10",
+  "found_count": 100,
+  "parsed_count": 98,
+  "new_count": 3,
+  "error_count": 2,
+  "dismissed_count": 1,
+  "processed_count": 100,
+  "progress_percent": 100.0,
+  "message": null,
+  "changes_detail_available": true,
+  "changes": {
+    "new": [],
+    "missing_from_source": [],
+    "dismissed": []
+  },
+  "errors": []
+}
+```
+
+Если запуск не найден:
+
+```json
+{
+  "error": "not_found"
+}
+```
+
+## Примеры curl
+
+Список tools:
+
+```bash
+curl http://localhost:8001/mcp \
+  -H "Content-Type: application/json" \
+  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}'
+```
+
+Поиск сотрудника:
+
+```bash
+curl http://localhost:8001/mcp \
+  -H "Content-Type: application/json" \
+  -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"search_employees","arguments":{"query":"Сергеев","limit":5}}}'
+```
+
+Полная синхронизация:
+
+```bash
+curl http://localhost:8001/mcp \
+  -H "Content-Type: application/json" \
+  -d '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"sync_employees","arguments":{"include_data":false}}}'
+```
+
+Delta-синхронизация:
+
+```bash
+curl http://localhost:8001/mcp \
+  -H "Content-Type: application/json" \
+  -d '{"jsonrpc":"2.0","id":4,"method":"tools/call","params":{"name":"sync_employees","arguments":{"client_hash":"known-sha256","include_data":true}}}'
+```
+
+## Как MCP используется клиентом
+
+1. Клиент вызывает `initialize` и проверяет `protocolVersion`.
+2. Клиент вызывает `tools/list`, чтобы получить актуальный список tools и input schemas.
+3. Для поиска и точечных запросов клиент вызывает `tools/call` с `search_employees`, `get_employee`, `list_employee_publications`, `list_employee_courses`, `get_crawl_status` или `get_crawl_run_details`.
+4. Для локального кэша клиент вызывает `get_service_info` или `sync_employees`.
+5. Клиент хранит последний `dataset.hash`.
+6. При следующей синхронизации клиент передает hash как `client_hash`.
+7. Сервер возвращает пустую delta, delta с изменениями или полный snapshot, если hash неизвестен.
+
+## Важные особенности реализации
+
+- MCP endpoint read-only: tools не запускают парсинг и не меняют сотрудников напрямую.
+- `get_service_info` и `sync_employees` могут создать новую запись `dataset_versions`, если состояние сотрудников изменилось и новой версии еще нет.
+- Все tool payloads возвращаются как JSON-строка внутри `content[0].text`.
+- `search_employees` ищет через `ilike` по `full_name` и `canonical_url`.
+- `get_employee` допускает частичный URL, поэтому строка `133709486` может найти `https://www.hse.ru/org/persons/133709486`.
+- Временные значения сериализуются через `isoformat()`, display-поля для админских payload формируются в часовом поясе `Europe/Moscow`.

README.md

@@ -6,10 +6,10 @@
 
 - `api`: FastAPI, REST API, HTML-админка, healthcheck.
 - `worker`: weekly scheduler, который запускает парсинг по `CRAWL_CRON`.
-- `mcp`: HTTP MCP endpoint с OAuth/OIDC access token для внешних агентов или legacy static token для локального режима.
+- `mcp`: открытый HTTP MCP endpoint для ИИ-агентов.
 - `postgres`: основная БД.
 
-Парсер использует фиксированный источник сотрудников, по умолчанию `https://miem.hse.ru/persons`. Для каждой карточки сохраняются ФИО, должности, год начала работы, контакты, идентификаторы, вкладки профиля, секции, публикации, курсы, ВКР, JSON-снапшот и сжатый HTML-снапшот. Ссылки обходятся только из меню профиля самого сотрудника (`person-menu`), например `#sci`, `#teaching`, `#main`.
+Парсер использует фиксированный источник сотрудников, по умолчанию `https://miem.hse.ru/persons`. Для каждой карточки сохраняются ФИО, должности, год начала работы, контакты, идентификаторы, вкладки профиля, секции, публикации, курсы, ВКР, JSON-снапшот и сжатый HTML-снапшот. Детальные публикации дополнительно нормализуются в отдельную таблицу `employee_publications`. Ссылки обходятся только из меню профиля самого сотрудника (`person-menu`), например `#sci`, `#teaching`, `#main`.
 
 ## Переменные окружения
 
@@ -27,13 +27,6 @@ cp .env.example .env
 - `CRAWL_LIMIT`: опциональный лимит профилей для тестового запуска.
 - `ADMIN_USERNAME`, `ADMIN_PASSWORD`: логин и пароль админки.
 - `SESSION_SECRET`: секрет подписи cookie.
-- `MCP_TOKEN`: статический bearer token для legacy/local режима `MCP_AUTH_MODE=token`.
-- `MCP_AUTH_MODE`: режим авторизации MCP: `oauth` для внешних агентов или `token` для локальной отладки.
-- `MCP_RESOURCE_URL`: публичный URL MCP endpoint, например `https://example.com/mcp`.
-- `MCP_OAUTH_ISSUER`: issuer внешнего OIDC-провайдера.
-- `MCP_OAUTH_AUDIENCE`: ожидаемый `aud` в OAuth access token.
-- `MCP_OAUTH_JWKS_URL`: JWKS endpoint; если не задан, используется `<issuer>/.well-known/jwks.json`.
-- `MCP_OAUTH_REQUIRED_SCOPE`: scope для доступа к MCP tools, по умолчанию `mcp:tools`.
 - `PARSER_USE_PLAYWRIGHT`: включение Playwright-рендера динамических вкладок.
 
 ## Локальный запуск
@@ -51,7 +44,6 @@ uvicorn app.main:app --reload
 
 - `Dashboard`: общая статистика, последний добавленный сотрудник, прогресс текущего/последнего парсинга и ручной запуск.
 - `Directory`: настраиваемая таблица сотрудников с фильтрами, сортировкой, пагинацией и выбором колонок.
-- `Employees`: простая legacy-таблица сотрудников.
 - `Runs`: история запусков, ошибки и progress bar.
 
 ## Docker Compose
@@ -66,7 +58,20 @@ docker compose up --build
 - MCP: `http://localhost:8001/mcp`
 - Postgres: `localhost:5432`
 
-Таблицы создаются приложением при старте. SQL-миграция для ручного применения лежит в `migrations/001_init.sql`.
+Таблицы создаются приложением при старте. При обновлении существующей базы приложение также добавляет недостающие runtime-колонки, например `crawl_runs.skipped_count`. SQL-миграции для ручного применения лежат в `migrations/`.
+
+## Наполнение БД
+
+Основная карточка сотрудника хранится в `employees`: профиль, статус, даты обнаружения/увольнения, текущий JSON `current_data`, checksum и версия парсера. История успешных изменений сохраняется в `employee_snapshots` вместе с JSON-снимком и сжатым HTML профиля.
+
+Публикации теперь хранятся в двух видах:
+
+- краткий список остается внутри `employees.current_data.sections[].publications` для обратной совместимости;
+- детальные записи сохраняются в `employee_publications` и связываются с сотрудником через `employee_id`.
+
+`employee_publications` содержит `publication_id`, название, год, тип публикации, язык, статус, ссылку на карточку HSE Publications, DOI, внешние/document-ссылки, citation text, аннотацию, описание, авторов, raw JSON ответа `searchPubs` и `source_hash` для безопасного повторного upsert. Уникальность поддерживается по `(employee_id, publication_id)` и `(employee_id, source_hash)`, поэтому повторный crawl не должен создавать дубликаты.
+
+`list_employee_publications` сначала читает `employee_publications`; если детальных строк еще нет, возвращает старые публикации из `current_data`.
 
 ## Парсинг
 
@@ -81,46 +86,39 @@ curl -X POST http://localhost:8000/api/crawl-runs --cookie "miem_admin_session=.
 - найденные сотрудники получают статус `active` и обновленный `last_seen_at`;
 - новые сотрудники добавляются в `employees`;
 - количество новых сотрудников за запуск сохраняется в `crawl_runs.new_count`;
+- публикации из HSE Publications записываются в `employee_publications`, а краткий список остается в JSON профиля;
 - активные сотрудники, исчезнувшие из текущего списка источника, получают статус `dismissed` и `dismissed_at`;
-- каждый успешный разбор сохраняет запись в `employee_snapshots`.
+- каждый успешный новый или измененный разбор сохраняет запись в `employee_snapshots`;
+- неизмененные профили учитываются в `crawl_runs.skipped_count` и не получают новый snapshot.
 
-Во время выполнения парсинга `found_count`, `parsed_count` и `error_count` обновляются в базе. Админка опрашивает `/api/crawl-runs/latest` и показывает прогресс как `parsed_count + error_count / found_count`.
+Во время выполнения парсинга `found_count`, `parsed_count`, `skipped_count` и `error_count` обновляются в базе. Админка опрашивает `/api/crawl-runs/latest` и показывает прогресс как `(parsed_count + skipped_count + error_count) / found_count`.
 
 ## MCP
 
-Endpoint: `POST /mcp`, авторизация `Authorization: Bearer <token>`.
+Endpoint: `POST /mcp`, без авторизации на уровне приложения.
-
-Для внешних ИИ-агентов используйте `MCP_AUTH_MODE=oauth`. В этом режиме статический `MCP_TOKEN` не принимается: клиент должен передать OAuth/OIDC access token с нужным scope.
 
 Поддерживаемые tools:
 
+- `get_service_info()`
+- `sync_employees(client_hash?, include_data?)`
 - `search_employees(query, status?, limit?)`
 - `get_employee(profile_id_or_url)`
-- `list_employee_publications(profile_id_or_url)`
+- `list_employee_publications(profile_id_or_url)` — публикации сотрудника; при наличии данных из `employee_publications` возвращает авторов, DOI, аннотацию, описание, citation text, год, тип, язык, статус и ссылку HSE Publications.
 - `list_employee_courses(profile_id_or_url)`
 - `get_crawl_status()`
+- `get_crawl_run_details(run_id)`
 
-Пример локального legacy-режима со статическим токеном:
+`get_service_info` возвращает метаданные сервиса, список tools и текущую версию набора сотрудников. `sync_employees` отдает полный snapshot или delta по `client_hash`; checksum набора строится по сотрудникам, их статусам и текущим checksums. Ответы tools возвращаются как JSON-строка внутри MCP `content[0].text`.
+
+Пример локального запроса списка tools:
 
 ```bash
 curl http://localhost:8001/mcp \
-  -H "Authorization: Bearer change-me-mcp-token" \
   -H "Content-Type: application/json" \
   -d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}'
 ```
 
-Для production OAuth/OIDC настройте внешний authorization server и включите режим `oauth`:
+Если MCP нужно ограничить, делайте это на сетевом уровне: localhost binding, VPN, firewall, reverse proxy или другой внешний контур доступа.
-
-```env
-MCP_AUTH_MODE=oauth
-MCP_RESOURCE_URL=https://example.com/mcp
-MCP_OAUTH_ISSUER=https://auth.example.com
-MCP_OAUTH_AUDIENCE=miem-mcp
-MCP_OAUTH_JWKS_URL=https://auth.example.com/.well-known/jwks.json
-MCP_OAUTH_REQUIRED_SCOPE=mcp:tools
-```
-
-MCP server работает как OAuth protected resource: он не выдает токены, а проверяет JWT access token по JWKS, `issuer`, `audience`, сроку действия и scope. Metadata для MCP-клиентов доступна по `GET /.well-known/oauth-protected-resource`.
 
 ## Обслуживание
 
@@ -131,4 +129,4 @@ docker compose exec postgres pg_dump -U miem miem_workers > backup.sql
 docker compose down
 ```
 
-Версия сервиса: `0.3.0`. Админка всегда показывает версии backend и frontend в footer.
+Версия сервиса: `0.6.2`. Админка всегда показывает версии backend и frontend в footer.

app/admin.py

@@ -8,8 +8,16 @@ from app.config import Settings, get_settings
 from app.db import SessionLocal, get_db
 from app.models import CrawlError, CrawlRun, Employee
 from app.security import SESSION_COOKIE, require_admin, sign_session, verify_admin
-from app.services.admin_data import employee_detail_payload, format_admin_datetime, list_employees_page, run_payload, stats_payload
+from app.services.admin_data import (
+    employee_detail_payload,
+    format_admin_datetime,
+    list_employees_page,
+    run_detail_payload,
+    run_payload,
+    stats_payload,
+)
 from app.services.crawl_control import get_running_run, run_crawl_if_idle
+from app.services.crawler import refresh_employee
 from app.version import BACKEND_VERSION, FRONTEND_VERSION
 
 router = APIRouter(prefix="/admin")
@@ -22,7 +30,7 @@ def dashboard(request: Request, db: Session = Depends(get_db), settings: Setting
     counts = stats_payload(db)
     counts["runs"] = db.scalar(select(func.count()).select_from(CrawlRun)) or 0
     counts["errors"] = db.scalar(select(func.count()).select_from(CrawlError)) or 0
-    run_models = db.scalars(select(CrawlRun).order_by(desc(CrawlRun.started_at)).limit(10)).all()
+    run_models = db.scalars(select(CrawlRun).order_by(desc(CrawlRun.started_at)).limit(5)).all()
     runs = [run_payload(run) for run in run_models]
     return _render(request, "dashboard.html", {"counts": counts, "runs": runs, "latest_run": runs[0] if runs else None})
 
@@ -137,10 +145,31 @@ def employee_detail(
     return _render(
         request,
         "employee_detail.html",
-        {"employee": employee, "employee_view": employee_detail_payload(employee), "snapshots": snapshots},
+        {
+            "employee": employee,
+            "employee_view": employee_detail_payload(employee),
+            "snapshots": snapshots,
+            "refresh_status": request.query_params.get("refresh_status"),
+        },
     )
 
 
+@router.post("/employees/{employee_id}/refresh")
+def refresh_employee_detail(
+    employee_id: int,
+    request: Request,
+    db: Session = Depends(get_db),
+    settings: Settings = Depends(get_settings),
+):
+    require_admin(request, settings)
+    employee = db.get(Employee, employee_id)
+    if not employee:
+        return RedirectResponse("/admin/directory", status_code=303)
+    run = refresh_employee(db, employee, settings)
+    status = "success" if run.status == "completed" else "error"
+    return RedirectResponse(f"/admin/employees/{employee_id}?refresh_status={status}", status_code=303)
+
+
 @router.get("/runs", response_class=HTMLResponse)
 def runs(request: Request, db: Session = Depends(get_db), settings: Settings = Depends(get_settings)):
     require_admin(request, settings)
@@ -150,6 +179,20 @@ def runs(request: Request, db: Session = Depends(get_db), settings: Settings = D
     return _render(request, "runs.html", {"runs": items, "errors": errors})
 
 
+@router.get("/runs/{run_id}", response_class=HTMLResponse)
+def run_detail(
+    run_id: int,
+    request: Request,
+    db: Session = Depends(get_db),
+    settings: Settings = Depends(get_settings),
+):
+    require_admin(request, settings)
+    run = db.get(CrawlRun, run_id)
+    if not run:
+        return RedirectResponse("/admin/runs", status_code=303)
+    return _render(request, "run_detail.html", {"run": run_detail_payload(db, run)})
+
+
 @router.post("/runs")
 def trigger_run(
     request: Request,

app/api.py

@@ -8,7 +8,7 @@ from app.config import Settings, get_settings
 from app.db import SessionLocal, get_db
 from app.models import CrawlRun, Employee
 from app.security import require_admin
-from app.services.admin_data import employee_display_payload, list_employees_page, run_payload, stats_payload
+from app.services.admin_data import employee_display_payload, list_employees_page, run_detail_payload, run_payload, stats_payload
 from app.services.crawl_control import get_running_run, run_crawl_if_idle
 from app.version import BACKEND_VERSION, FRONTEND_VERSION
 
@@ -88,6 +88,20 @@ def latest_crawl_run(
     return {"running": run_payload(running), "latest": run_payload(latest)}
 
 
+@router.get("/crawl-runs/{run_id}")
+def get_crawl_run(
+    run_id: int,
+    request: Request,
+    db: Session = Depends(get_db),
+    settings: Settings = Depends(get_settings),
+) -> dict:
+    require_admin(request, settings)
+    run = db.get(CrawlRun, run_id)
+    if not run:
+        return {"error": "not_found"}
+    return run_detail_payload(db, run) or {"error": "not_found"}
+
+
 @router.post("/crawl-runs")
 def trigger_crawl(
     request: Request,

app/config.py

@@ -1,6 +1,4 @@
 from functools import lru_cache
-from typing import Literal
-
 from pydantic import Field, field_validator
 from pydantic_settings import BaseSettings, SettingsConfigDict
 
@@ -19,13 +17,6 @@ class Settings(BaseSettings):
     admin_username: str = "admin"
     admin_password: str = "admin"
     session_secret: str = Field(default="dev-session-secret", min_length=8)
-    mcp_token: str = "dev-mcp-token"
-    mcp_auth_mode: Literal["token", "oauth"] = "oauth"
-    mcp_resource_url: str = "http://localhost:8001/mcp"
-    mcp_oauth_issuer: str = ""
-    mcp_oauth_audience: str = ""
-    mcp_oauth_jwks_url: str = ""
-    mcp_oauth_required_scope: str = "mcp:tools"
 
     @field_validator("crawl_limit", mode="before")
     @classmethod
@@ -34,15 +25,6 @@ class Settings(BaseSettings):
             return None
         return value
 
-    def oauth_jwks_url(self) -> str:
-        if self.mcp_oauth_jwks_url:
-            return self.mcp_oauth_jwks_url
-        issuer = self.mcp_oauth_issuer.rstrip("/")
-        if not issuer:
-            return ""
-        return f"{issuer}/.well-known/jwks.json"
-
-
 @lru_cache
 def get_settings() -> Settings:
     return Settings()

app/db.py

@@ -1,6 +1,6 @@
 from collections.abc import Generator
 
-from sqlalchemy import create_engine
+from sqlalchemy import create_engine, inspect, text
 from sqlalchemy.orm import DeclarativeBase, Session, sessionmaker
 
 from app.config import get_settings
@@ -25,6 +25,24 @@ def init_db() -> None:
     import app.models  # noqa: F401
 
     Base.metadata.create_all(bind=engine)
+    _ensure_runtime_schema()
+
+
+def _ensure_runtime_schema() -> None:
+    import app.models as models
+
+    inspector = inspect(engine)
+    table_names = set(inspector.get_table_names())
+    if "employees" in table_names and "employee_publications" not in table_names:
+        models.EmployeePublication.__table__.create(bind=engine, checkfirst=True)
+        inspector = inspect(engine)
+        table_names = set(inspector.get_table_names())
+    if "crawl_runs" not in table_names:
+        return
+    crawl_run_columns = {column["name"] for column in inspector.get_columns("crawl_runs")}
+    if "skipped_count" not in crawl_run_columns:
+        with engine.begin() as connection:
+            connection.execute(text("ALTER TABLE crawl_runs ADD COLUMN skipped_count INTEGER NOT NULL DEFAULT 0"))
 
 
 def get_db() -> Generator[Session, None, None]:

app/main.py

@@ -4,7 +4,6 @@ from fastapi.staticfiles import StaticFiles
 from app.admin import router as admin_router
 from app.api import router as api_router
 from app.db import init_db
-from app.mcp import metadata_router as mcp_metadata_router
 from app.mcp import router as mcp_router
 from app.version import BACKEND_VERSION
 
@@ -13,7 +12,6 @@ app.mount("/static", StaticFiles(directory="app/static"), name="static")
 app.include_router(api_router)
 app.include_router(admin_router)
 app.include_router(mcp_router)
-app.include_router(mcp_metadata_router)
 
 
 @app.on_event("startup")

app/mcp.py

@@ -4,16 +4,34 @@ from fastapi import APIRouter, Depends, Request
 from sqlalchemy import desc, or_, select
 from sqlalchemy.orm import Session
 
-from app.config import Settings, get_settings
 from app.db import get_db
-from app.models import CrawlRun, Employee
+from app.models import CrawlRun, Employee, EmployeePublication
-from app.security import mcp_protected_resource_metadata, require_mcp_auth
+from app.services.admin_data import run_detail_payload
+from app.services.dataset_versions import service_info_payload, sync_employees_payload
+from app.version import BACKEND_VERSION
 
 router = APIRouter(prefix="/mcp")
-metadata_router = APIRouter()
+PROTOCOL_VERSION = "2024-11-05"
+SERVICE_NAME = "miem-employees"
 
 
 TOOLS = [
+    {
+        "name": "get_service_info",
+        "description": "Return service metadata, supported tools, and current dataset version.",
+        "inputSchema": {"type": "object", "properties": {}},
+    },
+    {
+        "name": "sync_employees",
+        "description": "Synchronize employees by dataset hash. Returns a full snapshot or a delta from client_hash.",
+        "inputSchema": {
+            "type": "object",
+            "properties": {
+                "client_hash": {"type": "string"},
+                "include_data": {"type": "boolean", "default": True},
+            },
+        },
+    },
     {
         "name": "search_employees",
         "description": "Search MIEM employees by name or profile URL.",
@@ -34,7 +52,10 @@ TOOLS = [
     },
     {
         "name": "list_employee_publications",
-        "description": "List publications parsed from an employee profile.",
+        "description": (
+            "List employee publications with detailed fields when available: authors, DOI URL, annotation, "
+            "description, citation text, year, publication type, language, status, and HSE Publications URL."
+        ),
         "inputSchema": {"type": "object", "properties": {"profile_id_or_url": {"type": "string"}}, "required": ["profile_id_or_url"]},
     },
     {
@@ -47,6 +68,15 @@ TOOLS = [
         "description": "Return the latest crawl run status.",
         "inputSchema": {"type": "object", "properties": {}},
     },
+    {
+        "name": "get_crawl_run_details",
+        "description": "Return detailed employee changes and errors for one crawl run.",
+        "inputSchema": {
+            "type": "object",
+            "properties": {"run_id": {"type": "integer"}},
+            "required": ["run_id"],
+        },
+    },
 ]
 
 
@@ -54,9 +84,7 @@ TOOLS = [
 async def mcp_http(
     request: Request,
     db: Session = Depends(get_db),
-    settings: Settings = Depends(get_settings),
 ) -> dict:
-    require_mcp_auth(request, settings)
     payload = await request.json()
     method = payload.get("method")
     request_id = payload.get("id")
@@ -65,8 +93,8 @@ async def mcp_http(
     try:
         if method == "initialize":
             result = {
-                "protocolVersion": "2024-11-05",
+                "protocolVersion": PROTOCOL_VERSION,
-                "serverInfo": {"name": "miem-employees", "version": "0.1.0"},
+                "serverInfo": {"name": SERVICE_NAME, "version": BACKEND_VERSION},
                 "capabilities": {"tools": {}},
             }
         elif method == "tools/list":
@@ -81,6 +109,24 @@ async def mcp_http(
 
 
 def _call_tool(db: Session, name: str, arguments: dict) -> dict:
+    if name == "get_service_info":
+        return _tool_response(
+            service_info_payload(
+                db,
+                tools=TOOLS,
+                service_name=SERVICE_NAME,
+                backend_version=BACKEND_VERSION,
+                protocol_version=PROTOCOL_VERSION,
+            )
+        )
+    if name == "sync_employees":
+        return _tool_response(
+            sync_employees_payload(
+                db,
+                client_hash=arguments.get("client_hash"),
+                include_data=bool(arguments.get("include_data", True)),
+            )
+        )
     if name == "search_employees":
         return _tool_response(_search_employees(db, arguments))
     if name == "get_employee":
@@ -95,6 +141,9 @@ def _call_tool(db: Session, name: str, arguments: dict) -> dict:
     if name == "get_crawl_status":
         run = db.scalar(select(CrawlRun).order_by(desc(CrawlRun.started_at)).limit(1))
         return _tool_response(_run_payload(run) if run else {"status": "never_run"})
+    if name == "get_crawl_run_details":
+        run = db.get(CrawlRun, int(arguments["run_id"]))
+        return _tool_response(run_detail_payload(db, run) if run else {"error": "not_found"})
     raise ValueError(f"Unknown tool: {name}")
 
 
@@ -125,8 +174,14 @@ def _find_employee(db: Session, value: str) -> Employee | None:
 
 
 def _collect_section_items(employee: Employee | None, section_type: str) -> dict:
-    if not employee or not employee.current_data:
+    if not employee:
         return {"items": []}
+    if section_type == "publications":
+        publications = _stored_publications(employee)
+        if publications:
+            return {"employee": _employee_payload(employee, include_data=False), "items": publications}
+    if not employee.current_data:
+        return {"employee": _employee_payload(employee, include_data=False), "items": []}
     items = []
     for section in employee.current_data.get("sections") or []:
         if section.get("type") != section_type:
@@ -138,6 +193,41 @@ def _collect_section_items(employee: Employee | None, section_type: str) -> dict
     return {"employee": _employee_payload(employee, include_data=False), "items": items}
 
 
+def _stored_publications(employee: Employee) -> list[dict]:
+    return [_publication_payload(publication) for publication in sorted(employee.publications, key=_publication_sort_key)]
+
+
+def _publication_sort_key(publication: EmployeePublication) -> tuple:
+    return (publication.year or 0, publication.title or "", publication.id)
+
+
+def _publication_payload(publication: EmployeePublication) -> dict:
+    text = publication.citation_text or publication.title
+    payload = {
+        "id": publication.publication_id,
+        "publication_id": publication.publication_id,
+        "title": publication.title,
+        "text": text,
+        "url": publication.url,
+    }
+    optional = {
+        "year": publication.year,
+        "type": publication.publication_type,
+        "publication_type": publication.publication_type,
+        "language": publication.language,
+        "status": publication.status,
+        "doi_url": publication.doi_url,
+        "other_url": publication.other_url,
+        "document_url": publication.document_url,
+        "citation_text": publication.citation_text,
+        "annotation": publication.annotation,
+        "description": publication.description,
+        "authors": publication.authors,
+    }
+    payload.update({key: value for key, value in optional.items() if value not in (None, [], {})})
+    return payload
+
+
 def _employee_payload(employee: Employee, include_data: bool = True) -> dict:
     payload = {
         "profile_key": employee.profile_key,
@@ -162,6 +252,7 @@ def _run_payload(run: CrawlRun) -> dict:
         "finished_at": run.finished_at.isoformat() if run.finished_at else None,
         "found_count": run.found_count,
         "parsed_count": run.parsed_count,
+        "skipped_count": run.skipped_count,
         "error_count": run.error_count,
         "dismissed_count": run.dismissed_count,
     }
@@ -169,8 +260,3 @@ def _run_payload(run: CrawlRun) -> dict:
 
 def _tool_response(data: object) -> dict:
     return {"content": [{"type": "text", "text": json.dumps(data, ensure_ascii=False, default=str)}]}
-
-
-@metadata_router.get("/.well-known/oauth-protected-resource")
-def oauth_protected_resource(settings: Settings = Depends(get_settings)) -> dict:
-    return mcp_protected_resource_metadata(settings)

app/models.py

@@ -41,6 +41,8 @@ class Employee(Base):
 
     snapshots: Mapped[list["EmployeeSnapshot"]] = relationship(back_populates="employee")
     tabs: Mapped[list["ProfileTab"]] = relationship(back_populates="employee", cascade="all, delete-orphan")
+    publications: Mapped[list["EmployeePublication"]] = relationship(back_populates="employee", cascade="all, delete-orphan")
+    crawl_run_changes: Mapped[list["CrawlRunEmployeeChange"]] = relationship(back_populates="employee")
 
 
 class EmployeeSnapshot(Base):
@@ -59,6 +61,42 @@ class EmployeeSnapshot(Base):
     employee: Mapped[Employee] = relationship(back_populates="snapshots")
 
 
+class EmployeePublication(Base):
+    __tablename__ = "employee_publications"
+    __table_args__ = (
+        UniqueConstraint("employee_id", "publication_id", name="uq_employee_publications_employee_publication"),
+        UniqueConstraint("employee_id", "source_hash", name="uq_employee_publications_employee_source_hash"),
+        Index("ix_employee_publications_employee_id", "employee_id"),
+        Index("ix_employee_publications_publication_id", "publication_id"),
+        Index("ix_employee_publications_doi_url", "doi_url"),
+        Index("ix_employee_publications_year", "year"),
+        Index("ix_employee_publications_publication_type", "publication_type"),
+    )
+
+    id: Mapped[int] = mapped_column(Integer, primary_key=True)
+    employee_id: Mapped[int] = mapped_column(ForeignKey("employees.id", ondelete="CASCADE"), nullable=False)
+    publication_id: Mapped[str | None] = mapped_column(String(64))
+    title: Mapped[str] = mapped_column(Text, nullable=False)
+    year: Mapped[int | None] = mapped_column(Integer)
+    publication_type: Mapped[str | None] = mapped_column(String(64))
+    language: Mapped[str | None] = mapped_column(String(16))
+    status: Mapped[int | None] = mapped_column(Integer)
+    url: Mapped[str | None] = mapped_column(Text)
+    doi_url: Mapped[str | None] = mapped_column(Text)
+    other_url: Mapped[str | None] = mapped_column(Text)
+    document_url: Mapped[str | None] = mapped_column(Text)
+    citation_text: Mapped[str | None] = mapped_column(Text)
+    annotation: Mapped[dict | None] = mapped_column(json_type)
+    description: Mapped[dict | None] = mapped_column(json_type)
+    authors: Mapped[list | None] = mapped_column(json_type)
+    raw_data: Mapped[dict | None] = mapped_column(json_type)
+    source_hash: Mapped[str] = mapped_column(String(64), nullable=False)
+    created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow, nullable=False)
+    updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow, onupdate=utcnow, nullable=False)
+
+    employee: Mapped[Employee] = relationship(back_populates="publications")
+
+
 class CrawlRun(Base):
     __tablename__ = "crawl_runs"
 
@@ -69,11 +107,38 @@ class CrawlRun(Base):
     finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
     found_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
     parsed_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
+    skipped_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
     new_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
     error_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
     dismissed_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
     message: Mapped[str | None] = mapped_column(Text)
 
+    employee_changes: Mapped[list["CrawlRunEmployeeChange"]] = relationship(back_populates="crawl_run")
+    dataset_versions: Mapped[list["DatasetVersion"]] = relationship(back_populates="crawl_run")
+
+
+class CrawlRunEmployeeChange(Base):
+    __tablename__ = "crawl_run_employee_changes"
+    __table_args__ = (
+        Index("ix_crawl_run_employee_changes_run_id", "crawl_run_id"),
+        Index("ix_crawl_run_employee_changes_employee_id", "employee_id"),
+        Index("ix_crawl_run_employee_changes_change_type", "change_type"),
+    )
+
+    id: Mapped[int] = mapped_column(Integer, primary_key=True)
+    crawl_run_id: Mapped[int] = mapped_column(ForeignKey("crawl_runs.id"), nullable=False)
+    employee_id: Mapped[int | None] = mapped_column(ForeignKey("employees.id"))
+    profile_key: Mapped[str] = mapped_column(String(255), nullable=False)
+    profile_url: Mapped[str] = mapped_column(Text, nullable=False)
+    full_name: Mapped[str | None] = mapped_column(Text)
+    change_type: Mapped[str] = mapped_column(String(32), nullable=False)
+    profile_available: Mapped[bool | None] = mapped_column()
+    message: Mapped[str | None] = mapped_column(Text)
+    created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow, nullable=False)
+
+    crawl_run: Mapped[CrawlRun] = relationship(back_populates="employee_changes")
+    employee: Mapped[Employee | None] = relationship(back_populates="crawl_run_changes")
+
 
 class CrawlError(Base):
     __tablename__ = "crawl_errors"
@@ -108,3 +173,63 @@ class ParserSource(Base):
     source_url: Mapped[str] = mapped_column(Text, nullable=False)
     enabled: Mapped[bool] = mapped_column(default=True, nullable=False)
     created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow, nullable=False)
+
+
+class ParseResourceCache(Base):
+    __tablename__ = "parse_resource_cache"
+    __table_args__ = (
+        UniqueConstraint("profile_key", "resource_key", "request_fingerprint", name="uq_parse_resource_cache_resource"),
+        Index("ix_parse_resource_cache_profile_key", "profile_key"),
+    )
+
+    id: Mapped[int] = mapped_column(Integer, primary_key=True)
+    profile_key: Mapped[str] = mapped_column(String(255), nullable=False)
+    resource_key: Mapped[str] = mapped_column(String(255), nullable=False)
+    method: Mapped[str] = mapped_column(String(16), nullable=False)
+    url: Mapped[str] = mapped_column(Text, nullable=False)
+    request_fingerprint: Mapped[str] = mapped_column(String(64), nullable=False)
+    etag: Mapped[str | None] = mapped_column(Text)
+    last_modified: Mapped[str | None] = mapped_column(Text)
+    body_hash: Mapped[str] = mapped_column(String(64), nullable=False)
+    body_snapshot: Mapped[bytes] = mapped_column(LargeBinary, nullable=False)
+    parser_version: Mapped[str | None] = mapped_column(String(32))
+    fetched_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow, nullable=False)
+
+
+class DatasetVersion(Base):
+    __tablename__ = "dataset_versions"
+    __table_args__ = (
+        UniqueConstraint("hash", name="uq_dataset_versions_hash"),
+        Index("ix_dataset_versions_created_at", "created_at"),
+    )
+
+    id: Mapped[int] = mapped_column(Integer, primary_key=True)
+    hash: Mapped[str] = mapped_column(String(64), nullable=False)
+    previous_hash: Mapped[str | None] = mapped_column(String(64))
+    crawl_run_id: Mapped[int | None] = mapped_column(ForeignKey("crawl_runs.id"))
+    employee_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
+    active_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
+    dismissed_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
+    created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow, nullable=False)
+
+    crawl_run: Mapped[CrawlRun | None] = relationship(back_populates="dataset_versions")
+    items: Mapped[list["DatasetVersionItem"]] = relationship(back_populates="dataset_version", cascade="all, delete-orphan")
+
+
+class DatasetVersionItem(Base):
+    __tablename__ = "dataset_version_items"
+    __table_args__ = (
+        UniqueConstraint("dataset_version_id", "profile_key", name="uq_dataset_version_items_version_profile"),
+        Index("ix_dataset_version_items_hash", "dataset_version_id"),
+        Index("ix_dataset_version_items_profile_key", "profile_key"),
+    )
+
+    id: Mapped[int] = mapped_column(Integer, primary_key=True)
+    dataset_version_id: Mapped[int] = mapped_column(ForeignKey("dataset_versions.id"), nullable=False)
+    profile_key: Mapped[str] = mapped_column(String(255), nullable=False)
+    employee_id: Mapped[int | None] = mapped_column(ForeignKey("employees.id"))
+    status: Mapped[str] = mapped_column(String(32), nullable=False)
+    checksum: Mapped[str] = mapped_column(String(64), nullable=False)
+
+    dataset_version: Mapped[DatasetVersion] = relationship(back_populates="items")
+    employee: Mapped[Employee | None] = relationship()

app/parser/profile.py

@@ -1,3 +1,5 @@
+import hashlib
+import json
 import re
 from urllib.parse import urljoin
 
@@ -149,22 +151,42 @@ def parse_person_profile(
     headers: dict[str, str],
     timeout: int,
     use_playwright: bool = False,
+    resource_cache=None,
 ) -> dict | None:
     normalized_url = normalize_profile_url(source_url)
     if not normalized_url:
         return None
-    response = session.get(normalized_url, headers=headers, timeout=timeout)
+    profile_type, profile_id = parse_profile_identity(normalized_url)
-    response.raise_for_status()
+    cache_profile_key = f"{profile_type}:{profile_id}"
-    html = response.text
+    resource_manifest = []
+    html = _fetch_text(
+        session,
+        normalized_url,
+        headers,
+        timeout,
+        resource_cache=resource_cache,
+        profile_key=cache_profile_key,
+        resource_key="main-html",
+        resource_manifest=resource_manifest,
+    )
     if use_playwright:
         html = _render_with_playwright(normalized_url, html)
 
     soup = BeautifulSoup(html, "html.parser")
-    profile_type, profile_id = parse_profile_identity(normalized_url)
     header = extract_person_header(soup, normalized_url)
     tabs = extract_person_tabs(soup, normalized_url)
     sections = extract_sections(soup, normalized_url)
-    sections = enrich_sections_from_hse_widgets(session, soup, normalized_url, headers, timeout, sections)
+    sections = enrich_sections_from_hse_widgets(
+        session,
+        soup,
+        normalized_url,
+        headers,
+        timeout,
+        sections,
+        resource_cache=resource_cache,
+        profile_key=cache_profile_key,
+        resource_manifest=resource_manifest,
+    )
     internal_links = [tab["href"] for tab in tabs if tab.get("href")]
 
     return {
@@ -181,6 +203,7 @@ def parse_person_profile(
         "employee_internal_links": internal_links,
         "parser_version": BACKEND_VERSION,
         "_html": html,
+        "_resource_manifest": resource_manifest,
     }
 
 
@@ -191,13 +214,33 @@ def enrich_sections_from_hse_widgets(
     headers: dict[str, str],
     timeout: int,
     sections: list[dict],
+    resource_cache=None,
+    profile_key: str | None = None,
+    resource_manifest: list[dict] | None = None,
 ) -> list[dict]:
     enriched = list(sections)
-    publications = _load_widget_publications(session, soup, headers, timeout)
+    publications = _load_widget_publications(
+        session,
+        soup,
+        headers,
+        timeout,
+        resource_cache=resource_cache,
+        profile_key=profile_key,
+        resource_manifest=resource_manifest,
+    )
     if publications:
         enriched = _upsert_publications_section(enriched, publications)
 
-    theses = _load_widget_graduation_theses(session, soup, source_url, headers, timeout)
+    theses = _load_widget_graduation_theses(
+        session,
+        soup,
+        source_url,
+        headers,
+        timeout,
+        resource_cache=resource_cache,
+        profile_key=profile_key,
+        resource_manifest=resource_manifest,
+    )
     if theses:
         enriched = _upsert_graduation_theses_section(enriched, theses)
     return enriched
@@ -226,7 +269,16 @@ def _render_with_playwright(source_url: str, fallback_html: str) -> str:
         return fallback_html
 
 
-def _load_widget_publications(session: Session, soup: BeautifulSoup, headers: dict[str, str], timeout: int) -> list[dict]:
+def _load_widget_publications(
+    session: Session,
+    soup: BeautifulSoup,
+    headers: dict[str, str],
+    timeout: int,
+    *,
+    resource_cache=None,
+    profile_key: str | None = None,
+    resource_manifest: list[dict] | None = None,
+) -> list[dict]:
     script = soup.select_one('script[data-widget-name="AuthorSearch"][data-author]')
     if not script:
         return []
@@ -251,22 +303,37 @@ def _load_widget_publications(session: Session, soup: BeautifulSoup, headers: di
             },
         }
         try:
-            response = session.post(
+            if resource_cache and profile_key:
-                "https://publications.hse.ru/api/searchPubs",
+                text = _fetch_text(
-                json=payload,
+                    session,
-                headers=headers,
+                    "https://publications.hse.ru/api/searchPubs",
-                timeout=timeout,
+                    headers,
-            )
+                    timeout,
-            response.raise_for_status()
+                    resource_cache=resource_cache,
-            data = response.json()
+                    profile_key=profile_key,
+                    resource_key=f"publications-page-{page_id}",
+                    resource_manifest=resource_manifest,
+                    method="POST",
+                    json_payload=payload,
+                )
+                data = json.loads(text)
+            else:
+                response = session.post(
+                    "https://publications.hse.ru/api/searchPubs",
+                    json=payload,
+                    headers=headers,
+                    timeout=timeout,
+                )
+                response.raise_for_status()
+                data = response.json()
         except Exception:
             return publications
 
         result = data.get("result") if isinstance(data, dict) else {}
-        items = result.get("items") if isinstance(result, dict) else []
+        items = _extract_publication_items(result)
-        if not isinstance(items, list) or not items:
+        if not items:
             break
-        publications.extend(_normalize_publication_item(item) for item in items if isinstance(item, dict))
+        publications.extend(_normalize_publication_item(item, author_id) for item in items)
 
         total = int(result.get("total") or 0)
         if not result.get("more") and len(publications) >= total:
@@ -275,12 +342,44 @@ def _load_widget_publications(session: Session, soup: BeautifulSoup, headers: di
     return _dedupe_publications(publications)
 
 
+def _extract_publication_items(result: object) -> list[dict]:
+    if not isinstance(result, dict):
+        return []
+    return _flatten_publication_items(result.get("items"))
+
+
+def _flatten_publication_items(value: object) -> list[dict]:
+    if isinstance(value, list):
+        return [item for item in value if _is_publication_item(item)]
+    if not isinstance(value, dict):
+        return []
+
+    nested_items = value.get("items")
+    if isinstance(nested_items, list):
+        return [item for item in nested_items if _is_publication_item(item)]
+    if isinstance(nested_items, dict):
+        return _flatten_publication_items(nested_items)
+
+    publications = []
+    for child in value.values():
+        publications.extend(_flatten_publication_items(child))
+    return publications
+
+
+def _is_publication_item(value: object) -> bool:
+    return isinstance(value, dict) and ("id" in value or "title" in value)
+
+
 def _load_widget_graduation_theses(
     session: Session,
     soup: BeautifulSoup,
     source_url: str,
     headers: dict[str, str],
     timeout: int,
+    *,
+    resource_cache=None,
+    profile_key: str | None = None,
+    resource_manifest: list[dict] | None = None,
 ) -> list[dict]:
     script = soup.select_one('script[src*="/n/stat/vkr/app.js"][data-person-id]')
     if not script:
@@ -292,14 +391,30 @@ def _load_widget_graduation_theses(
 
     request_headers = {**headers, "x-portal-language": "ru"}
     try:
-        response = session.get(
+        url = urljoin(source_url, api_url)
-            urljoin(source_url, api_url),
+        params = {"supervisorId": person_id}
-            params={"supervisorId": person_id},
+        if resource_cache and profile_key:
-            headers=request_headers,
+            text = _fetch_text(
-            timeout=timeout,
+                session,
-        )
+                url,
-        response.raise_for_status()
+                request_headers,
-        data = response.json()
+                timeout,
+                resource_cache=resource_cache,
+                profile_key=profile_key,
+                resource_key="graduation-theses",
+                resource_manifest=resource_manifest,
+                params=params,
+            )
+            data = json.loads(text)
+        else:
+            response = session.get(
+                url,
+                params=params,
+                headers=request_headers,
+                timeout=timeout,
+            )
+            response.raise_for_status()
+            data = response.json()
     except Exception:
         return []
 
@@ -359,7 +474,7 @@ def _infer_section_type(title: str, nodes: list) -> str:
     lowered = title.lower()
     if _has_table(nodes):
         return "table"
-    if "публикац" in lowered:
+    if _is_publications_title(lowered):
         return "publications"
     if "учебные курсы" in lowered:
         return "courses_by_year"
@@ -370,6 +485,10 @@ def _infer_section_type(title: str, nodes: list) -> str:
     return "generic"
 
 
+def _is_publications_title(lowered_title: str) -> bool:
+    return lowered_title.startswith("публикац")
+
+
 def _has_table(nodes: list) -> bool:
     return any(isinstance(node, Tag) and (node.name == "table" or node.find("table")) for node in nodes)
 
@@ -456,20 +575,37 @@ def _parse_vkr_items(nodes: list) -> list[str]:
     return [item for item in dict.fromkeys(items) if item]
 
 
-def _normalize_publication_item(item: dict) -> dict:
+def _normalize_publication_item(item: dict, current_author_id: str | None = None) -> dict:
     publication_id = str(item.get("id") or "").strip()
     title = _html_to_text(item.get("title"))
-    year = item.get("year")
+    year = _int_or_none(item.get("year"))
     publication_type = str(item.get("type") or "").strip() or None
     description = item.get("description") if isinstance(item.get("description"), dict) else {}
     short_description = _localized_value(description.get("short")) or _localized_value(description.get("shortLeft"))
+    documents = item.get("documents") if isinstance(item.get("documents"), dict) else {}
+    language = item.get("language") if isinstance(item.get("language"), dict) else {}
+    annotation = _localized_text_map(item.get("annotation"))
+    authors = _normalize_publication_authors(item.get("authorsByType"), current_author_id)
+    citation_text = normalize_ws(str(description.get("main") or "")) or _build_publication_citation(title, authors, year)
     text = normalize_ws(" ".join(part for part in [title, str(year or ""), short_description] if part))
     return {
         "id": publication_id or None,
+        "publication_id": publication_id or None,
         "title": title or publication_id,
         "year": year,
         "type": publication_type,
+        "publication_type": publication_type,
+        "language": normalize_ws(language.get("name")) or None,
+        "status": _int_or_none(item.get("status")),
         "url": f"https://publications.hse.ru/view/{publication_id}" if publication_id else None,
+        "doi_url": _document_href(documents, "DOI"),
+        "other_url": _document_href(documents, "OTHER_URL"),
+        "document_url": _document_href(documents, "DOCUMENT"),
+        "citation_text": citation_text or None,
+        "annotation": annotation,
+        "description": description or None,
+        "authors": authors,
+        "raw_data": item,
         "text": text or title or publication_id,
     }
 
@@ -566,12 +702,69 @@ def _html_to_text(value: object) -> str:
     return normalize_ws(BeautifulSoup(str(value or ""), "html.parser").get_text(" ", strip=True))
 
 
+def _localized_text_map(value: object) -> dict[str, str]:
+    if not isinstance(value, dict):
+        return {}
+    localized = {}
+    for key in ("ru", "en", "publ"):
+        text = _html_to_text(value.get(key))
+        if text:
+            localized[key] = text
+    return localized
+
+
 def _localized_value(value: object) -> str:
     if isinstance(value, dict):
         return normalize_ws(value.get("ru") or value.get("publ") or value.get("en"))
     return normalize_ws(str(value or ""))
 
 
+def _normalize_publication_authors(value: object, current_author_id: str | None) -> list[dict]:
+    if not isinstance(value, dict):
+        return []
+    authors = []
+    for author in value.get("author") or []:
+        if not isinstance(author, dict):
+            continue
+        title = author.get("title") if isinstance(author.get("title"), dict) else {}
+        reverse_title = author.get("reverseTitle") if isinstance(author.get("reverseTitle"), dict) else {}
+        author_id = normalize_ws(author.get("id"))
+        href = normalize_ws(author.get("href"))
+        authors.append(
+            {
+                "id": author_id or None,
+                "href": urljoin("https://www.hse.ru", href) if href else None,
+                "title_ru": _html_to_text(title.get("ru")),
+                "title_en": _html_to_text(title.get("en")),
+                "reverse_title_ru": _html_to_text(reverse_title.get("ru")),
+                "reverse_title_en": _html_to_text(reverse_title.get("en")),
+                "alt_name": normalize_ws(author.get("altName")) or None,
+                "other_name": normalize_ws(author.get("otherName")) or None,
+                "is_current_employee": bool(current_author_id and author_id == current_author_id),
+            }
+        )
+    return authors
+
+
+def _document_href(documents: dict, key: str) -> str | None:
+    document = documents.get(key)
+    if not isinstance(document, dict):
+        return None
+    return normalize_ws(document.get("href")) or None
+
+
+def _build_publication_citation(title: str, authors: list[dict], year: int | None) -> str:
+    author_names = [author.get("title_ru") or author.get("title_en") or author.get("alt_name") for author in authors]
+    return normalize_ws(". ".join(part for part in [", ".join(filter(None, author_names)), title, str(year or "")] if part))
+
+
+def _int_or_none(value: object) -> int | None:
+    try:
+        return int(value)
+    except (TypeError, ValueError):
+        return None
+
+
 def _slugify(value: str) -> str:
     cleaned = re.sub(r"[^\w\s-]", "", value.lower(), flags=re.UNICODE)
     return re.sub(r"[-\s]+", "_", cleaned).strip("_") or "section"
@@ -597,3 +790,62 @@ def _dedupe_dicts(items: list[dict]) -> list[dict]:
             seen.add(key)
             unique.append(item)
     return unique
+
+
+def _fetch_text(
+    session: Session,
+    url: str,
+    headers: dict[str, str],
+    timeout: int,
+    *,
+    resource_cache=None,
+    profile_key: str | None = None,
+    resource_key: str,
+    resource_manifest: list[dict] | None,
+    method: str = "GET",
+    json_payload: object | None = None,
+    params: dict | None = None,
+) -> str:
+    if resource_cache and profile_key:
+        cached = resource_cache.fetch_text(
+            session,
+            profile_key=profile_key,
+            resource_key=resource_key,
+            method=method,
+            url=url,
+            headers=headers,
+            timeout=timeout,
+            json_payload=json_payload,
+            params=params,
+        )
+        if resource_manifest is not None:
+            resource_manifest.append(
+                {
+                    "resource_key": resource_key,
+                    "method": method,
+                    "url": url,
+                    "body_hash": cached.body_hash,
+                    "from_cache": cached.from_cache,
+                    "status_code": cached.status_code,
+                }
+            )
+        return cached.text
+
+    if method.upper() == "POST":
+        response = session.post(url, json=json_payload, headers=headers, timeout=timeout, params=params)
+    else:
+        response = session.get(url, headers=headers, timeout=timeout, params=params)
+    response.raise_for_status()
+    text = response.text
+    if resource_manifest is not None:
+        resource_manifest.append(
+            {
+                "resource_key": resource_key,
+                "method": method,
+                "url": url,
+                "body_hash": hashlib.sha256(text.encode("utf-8")).hexdigest(),
+                "from_cache": False,
+                "status_code": response.status_code,
+            }
+        )
+    return text

app/security.py

@@ -3,10 +3,7 @@ import hashlib
 import hmac
 import json
 import time
-from functools import lru_cache
 
-import jwt
-from jwt import PyJWKClient, PyJWTError
 from fastapi import HTTPException, Request, status
 
 from app.config import Settings
@@ -47,93 +44,3 @@ def require_admin(request: Request, settings: Settings) -> str:
     if not username:
         raise HTTPException(status_code=status.HTTP_303_SEE_OTHER, headers={"Location": "/admin/login"})
     return username
-
-
-def require_mcp_auth(request: Request, settings: Settings) -> None:
-    auth = request.headers.get("authorization", "")
-    if not auth.startswith("Bearer "):
-        raise _mcp_unauthorized(settings, "Missing bearer token")
-
-    token = auth.removeprefix("Bearer ").strip()
-    if _mcp_static_token_allowed(settings) and hmac.compare_digest(token, settings.mcp_token):
-        return
-    if _mcp_oauth_allowed(settings):
-        _validate_mcp_oauth_token(token, settings)
-        return
-    raise _mcp_unauthorized(settings, "Invalid MCP token")
-
-
-def require_mcp_token(request: Request, settings: Settings) -> None:
-    require_mcp_auth(request, settings)
-
-
-def mcp_protected_resource_metadata(settings: Settings) -> dict:
-    authorization_servers = [settings.mcp_oauth_issuer.rstrip("/")] if settings.mcp_oauth_issuer else []
-    return {
-        "resource": settings.mcp_resource_url,
-        "authorization_servers": authorization_servers,
-        "bearer_methods_supported": ["header"],
-        "scopes_supported": [settings.mcp_oauth_required_scope],
-        "resource_documentation": settings.mcp_resource_url,
-    }
-
-
-def _mcp_static_token_allowed(settings: Settings) -> bool:
-    return settings.mcp_auth_mode == "token"
-
-
-def _mcp_oauth_allowed(settings: Settings) -> bool:
-    return settings.mcp_auth_mode == "oauth"
-
-
-def _validate_mcp_oauth_token(token: str, settings: Settings) -> None:
-    if not settings.mcp_oauth_issuer or not settings.mcp_oauth_audience or not settings.oauth_jwks_url():
-        raise _mcp_unauthorized(settings, "MCP OAuth is not configured")
-    try:
-        signing_key = _get_mcp_oauth_signing_key(token, settings).key
-        claims = jwt.decode(
-            token,
-            signing_key,
-            algorithms=["RS256", "RS384", "RS512", "ES256", "ES384", "ES512"],
-            audience=settings.mcp_oauth_audience,
-            issuer=settings.mcp_oauth_issuer.rstrip("/"),
-        )
-    except PyJWTError as exc:
-        raise _mcp_unauthorized(settings, "Invalid OAuth access token") from exc
-    if not _claims_have_scope(claims, settings.mcp_oauth_required_scope):
-        raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Missing required MCP OAuth scope")
-
-
-def _claims_have_scope(claims: dict, required_scope: str) -> bool:
-    scopes: set[str] = set()
-    scope = claims.get("scope")
-    if isinstance(scope, str):
-        scopes.update(scope.split())
-    scp = claims.get("scp")
-    if isinstance(scp, str):
-        scopes.update(scp.split())
-    elif isinstance(scp, list):
-        scopes.update(str(item) for item in scp)
-    return required_scope in scopes
-
-
-@lru_cache(maxsize=16)
-def _get_jwk_client(jwks_url: str) -> PyJWKClient:
-    return PyJWKClient(jwks_url)
-
-
-def _get_mcp_oauth_signing_key(token: str, settings: Settings):
-    return _get_jwk_client(settings.oauth_jwks_url()).get_signing_key_from_jwt(token)
-
-
-def _mcp_unauthorized(settings: Settings, detail: str) -> HTTPException:
-    headers = {}
-    if _mcp_oauth_allowed(settings):
-        headers["WWW-Authenticate"] = f'Bearer resource_metadata="{_mcp_metadata_url(settings)}"'
-    return HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail=detail, headers=headers)
-
-
-def _mcp_metadata_url(settings: Settings) -> str:
-    resource_url = settings.mcp_resource_url.rstrip("/")
-    base_url = resource_url[: -len("/mcp")] if resource_url.endswith("/mcp") else resource_url
-    return f"{base_url}/.well-known/oauth-protected-resource"

app/services/admin_data.py

@@ -8,7 +8,7 @@ from zoneinfo import ZoneInfo
 from sqlalchemy import Select, Text, and_, desc, func, or_, select
 from sqlalchemy.orm import Session
 
-from app.models import CrawlRun, Employee
+from app.models import CrawlError, CrawlRun, CrawlRunEmployeeChange, Employee
 
 EMPLOYEE_SORTS = {
     "full_name": Employee.full_name,
@@ -153,7 +153,7 @@ def stats_payload(db: Session) -> dict[str, Any]:
 def run_payload(run: CrawlRun | None) -> dict[str, Any] | None:
     if not run:
         return None
-    processed = run.parsed_count + run.error_count
+    processed = run.parsed_count + run.skipped_count + run.error_count
     percent = round((processed / run.found_count) * 100, 1) if run.found_count else 0
     return {
         "id": run.id,
@@ -166,6 +166,7 @@ def run_payload(run: CrawlRun | None) -> dict[str, Any] | None:
         "finished_display": format_admin_datetime(run.finished_at),
         "found_count": run.found_count,
         "parsed_count": run.parsed_count,
+        "skipped_count": run.skipped_count,
         "new_count": run.new_count,
         "error_count": run.error_count,
         "dismissed_count": run.dismissed_count,
@@ -175,6 +176,26 @@ def run_payload(run: CrawlRun | None) -> dict[str, Any] | None:
     }
 
 
+def run_detail_payload(db: Session, run: CrawlRun | None) -> dict[str, Any] | None:
+    if not run:
+        return None
+    changes = db.scalars(
+        select(CrawlRunEmployeeChange)
+        .where(CrawlRunEmployeeChange.crawl_run_id == run.id)
+        .order_by(CrawlRunEmployeeChange.created_at, CrawlRunEmployeeChange.id)
+    ).all()
+    errors = db.scalars(select(CrawlError).where(CrawlError.crawl_run_id == run.id).order_by(CrawlError.created_at)).all()
+    grouped_changes = {"new": [], "missing_from_source": [], "dismissed": []}
+    for change in changes:
+        grouped_changes.setdefault(change.change_type, []).append(_change_payload(change))
+    return {
+        **(run_payload(run) or {}),
+        "changes_detail_available": bool(changes),
+        "changes": grouped_changes,
+        "errors": [_crawl_error_payload(error) for error in errors],
+    }
+
+
 def format_admin_datetime(value: Any) -> str:
     if not value:
         return "Не указано"
@@ -200,6 +221,52 @@ def _run_status_display(status: str | None) -> str:
     return labels.get(status or "", status or "Не указано")
 
 
+def _change_payload(change: CrawlRunEmployeeChange) -> dict[str, Any]:
+    return {
+        "id": change.id,
+        "employee_id": change.employee_id,
+        "profile_key": change.profile_key,
+        "profile_url": change.profile_url,
+        "full_name": change.full_name,
+        "change_type": change.change_type,
+        "change_type_display": _change_type_display(change.change_type),
+        "profile_available": change.profile_available,
+        "profile_available_display": _profile_available_display(change.profile_available),
+        "message": change.message,
+        "created_at": change.created_at.isoformat() if change.created_at else None,
+        "created_display": format_admin_datetime(change.created_at),
+    }
+
+
+def _crawl_error_payload(error: CrawlError) -> dict[str, Any]:
+    return {
+        "id": error.id,
+        "crawl_run_id": error.crawl_run_id,
+        "profile_url": error.profile_url,
+        "error_type": error.error_type,
+        "message": error.message,
+        "created_at": error.created_at.isoformat() if error.created_at else None,
+        "created_display": format_admin_datetime(error.created_at),
+    }
+
+
+def _change_type_display(change_type: str | None) -> str:
+    labels = {
+        "new": "Новый",
+        "missing_from_source": "Потеряшка",
+        "dismissed": "Уволен",
+    }
+    return labels.get(change_type or "", change_type or "Не указано")
+
+
+def _profile_available_display(value: bool | None) -> str:
+    if value is True:
+        return "Профиль доступен"
+    if value is False:
+        return "Профиль недоступен"
+    return "Не проверялось"
+
+
 def _count_section_items(sections: list[dict[str, Any]], section_type: str) -> int:
     total = 0
     for section in sections:

app/services/crawler.py

@@ -1,18 +1,30 @@
 import gzip
 import hashlib
 import json
+import re
 import time
 from datetime import datetime, timezone
 
 import requests
-from sqlalchemy import select
+from sqlalchemy import inspect, select
 from sqlalchemy.orm import Session
 
 from app.config import Settings
-from app.models import CrawlError, CrawlRun, Employee, EmployeeSnapshot, ParserSource, ProfileTab
+from app.models import (
+    CrawlError,
+    CrawlRun,
+    CrawlRunEmployeeChange,
+    Employee,
+    EmployeePublication,
+    EmployeeSnapshot,
+    ParserSource,
+    ProfileTab,
+)
 from app.parser.collector import collect_profile_links
 from app.parser.profile import parse_person_profile
 from app.parser.profile_url import profile_key
+from app.services.dataset_versions import get_or_create_current_version
+from app.services.resource_cache import ResourceCache
 
 HEADERS = {
     "User-Agent": "Mozilla/5.0 (compatible; MIEMEmployeesBot/0.1.0; +https://miem.hse.ru/)"
@@ -28,8 +40,10 @@ def run_crawl(db: Session, settings: Settings) -> CrawlRun:
 
     found_keys: set[str] = set()
     parsed_count = 0
+    skipped_count = 0
     try:
         with requests.Session() as session:
+            resource_cache = ResourceCache(db)
             urls = collect_profile_links(session, source.source_url, HEADERS, settings.request_timeout)
             if settings.crawl_limit:
                 urls = urls[: settings.crawl_limit]
@@ -47,12 +61,17 @@ def run_crawl(db: Session, settings: Settings) -> CrawlRun:
                         HEADERS,
                         settings.request_timeout,
                         settings.parser_use_playwright,
+                        resource_cache=resource_cache,
                     )
                     if not parsed:
                         continue
-                    _upsert_employee(db, run, parsed)
+                    _, changed = _upsert_employee(db, run, parsed)
-                    parsed_count += 1
+                    if changed:
+                        parsed_count += 1
+                    else:
+                        skipped_count += 1
                     run.parsed_count = parsed_count
+                    run.skipped_count = skipped_count
                     db.commit()
                 except Exception as exc:
                     run.error_count += 1
@@ -68,8 +87,9 @@ def run_crawl(db: Session, settings: Settings) -> CrawlRun:
                 finally:
                     time.sleep(settings.request_delay_seconds)
 
-        run.dismissed_count = _mark_dismissed(db, found_keys)
+            run.dismissed_count = _mark_dismissed(db, run, found_keys, session, settings.request_timeout)
         run.status = "completed"
+        get_or_create_current_version(db, crawl_run_id=run.id)
     except Exception as exc:
         run.status = "failed"
         run.message = str(exc)
@@ -80,6 +100,54 @@ def run_crawl(db: Session, settings: Settings) -> CrawlRun:
     return run
 
 
+def refresh_employee(db: Session, employee: Employee, settings: Settings) -> CrawlRun:
+    run = CrawlRun(source_url=employee.canonical_url, status="running", found_count=1)
+    db.add(run)
+    db.commit()
+    db.refresh(run)
+
+    try:
+        with requests.Session() as session:
+            resource_cache = ResourceCache(db)
+            parsed = parse_person_profile(
+                session,
+                employee.canonical_url,
+                HEADERS,
+                settings.request_timeout,
+                settings.parser_use_playwright,
+                resource_cache=resource_cache,
+            )
+        if not parsed:
+            raise ValueError("Профиль не удалось распарсить.")
+        if _parsed_profile_key(parsed) != employee.profile_key:
+            raise ValueError("Распарсенный профиль не совпадает с обновляемым сотрудником.")
+
+        _, changed = _upsert_employee(db, run, parsed)
+        if changed:
+            run.parsed_count = 1
+        else:
+            run.skipped_count = 1
+        run.status = "completed"
+        get_or_create_current_version(db, crawl_run_id=run.id)
+    except Exception as exc:
+        run.status = "failed"
+        run.error_count = 1
+        run.message = str(exc)
+        db.add(
+            CrawlError(
+                crawl_run_id=run.id,
+                profile_url=employee.canonical_url,
+                error_type=type(exc).__name__,
+                message=str(exc),
+            )
+        )
+    finally:
+        run.finished_at = datetime.now(timezone.utc)
+        db.commit()
+        db.refresh(run)
+    return run
+
+
 def _ensure_source(db: Session, source_url: str) -> ParserSource:
     source = db.scalar(select(ParserSource).where(ParserSource.source_url == source_url))
     if source:
@@ -91,10 +159,15 @@ def _ensure_source(db: Session, source_url: str) -> ParserSource:
     return source
 
 
-def _upsert_employee(db: Session, run: CrawlRun, parsed: dict) -> Employee:
+def _parsed_profile_key(parsed: dict) -> str:
+    return f"{parsed.get('profile_type')}:{parsed.get('profile_id')}"
+
+
+def _upsert_employee(db: Session, run: CrawlRun, parsed: dict) -> tuple[Employee, bool]:
     html = parsed.pop("_html", None)
+    parsed.pop("_resource_manifest", None)
     checksum = _checksum(parsed)
-    key = f"{parsed.get('profile_type')}:{parsed.get('profile_id')}"
+    key = _parsed_profile_key(parsed)
     employee = db.scalar(select(Employee).where(Employee.profile_key == key))
     now = datetime.now(timezone.utc)
     if not employee:
@@ -107,54 +180,257 @@ def _upsert_employee(db: Session, run: CrawlRun, parsed: dict) -> Employee:
         )
         db.add(employee)
         run.new_count += 1
+        is_new = True
+    else:
+        is_new = False
 
+    parser_version = parsed.get("parser_version")
+    changed = is_new or employee.current_checksum != checksum or employee.parser_version != parser_version
     employee.full_name = parsed.get("full_name")
     employee.status = "active"
     employee.last_seen_at = now
     employee.dismissed_at = None
-    employee.parser_version = parsed.get("parser_version")
+    employee.parser_version = parser_version
-    employee.current_data = parsed
+    if changed:
+        employee.current_data = parsed
     employee.current_checksum = checksum
     db.flush()
 
-    db.query(ProfileTab).filter(ProfileTab.employee_id == employee.id).delete()
+    if is_new:
-    for tab in parsed.get("tabs") or []:
+        _record_employee_change(
+            db,
+            run,
+            employee,
+            "new",
+            profile_available=True,
+            message="Сотрудник впервые найден в источнике.",
+        )
+
+    if changed:
+        db.query(ProfileTab).filter(ProfileTab.employee_id == employee.id).delete()
+        for tab in parsed.get("tabs") or []:
+            db.add(
+                ProfileTab(
+                    employee_id=employee.id,
+                    title=tab.get("title") or "",
+                    href=tab.get("href") or "",
+                    data_index=tab.get("data_index"),
+                )
+            )
+
         db.add(
-            ProfileTab(
+            EmployeeSnapshot(
                 employee_id=employee.id,
-                title=tab.get("title") or "",
+                crawl_run_id=run.id,
-                href=tab.get("href") or "",
+                parsed_data=parsed,
-                data_index=tab.get("data_index"),
+                html_snapshot=gzip.compress(html.encode("utf-8")) if html else None,
+                checksum=checksum,
+                parser_version=parser_version,
+            )
+        )
+    db.flush()
+    _try_sync_employee_publications(db, run, employee, parsed)
+    return employee, changed
+
+
+def _try_sync_employee_publications(db: Session, run: CrawlRun, employee: Employee, parsed: dict) -> None:
+    try:
+        if not _publication_payloads(parsed):
+            return
+        if not _employee_publications_table_exists(db):
+            return
+        with db.begin_nested():
+            _sync_employee_publications(db, employee, parsed)
+    except Exception as exc:
+        db.add(
+            CrawlError(
+                crawl_run_id=run.id,
+                profile_url=employee.canonical_url,
+                error_type=type(exc).__name__,
+                message=f"Не удалось сохранить публикации сотрудника: {exc}",
             )
         )
 
-    db.add(
+
-        EmployeeSnapshot(
+def _employee_publications_table_exists(db: Session) -> bool:
-            employee_id=employee.id,
+    return inspect(db.connection()).has_table(EmployeePublication.__tablename__)
-            crawl_run_id=run.id,
-            parsed_data=parsed,
-            html_snapshot=gzip.compress(html.encode("utf-8")) if html else None,
-            checksum=checksum,
-            parser_version=parsed.get("parser_version"),
-        )
-    )
-    return employee
 
 
-def _mark_dismissed(db: Session, found_keys: set[str]) -> int:
+def _sync_employee_publications(db: Session, employee: Employee, parsed: dict) -> None:
+    publications = _publication_payloads(parsed)
+    seen_hashes = set()
+    for publication in publications:
+        source_hash = _publication_hash(publication)
+        seen_hashes.add(source_hash)
+        publication_id = _clean_optional(publication.get("publication_id") or publication.get("id"))
+        existing = None
+        if publication_id:
+            existing = db.scalar(
+                select(EmployeePublication).where(
+                    EmployeePublication.employee_id == employee.id,
+                    EmployeePublication.publication_id == publication_id,
+                )
+            )
+        if not existing:
+            existing = db.scalar(
+                select(EmployeePublication).where(
+                    EmployeePublication.employee_id == employee.id,
+                    EmployeePublication.source_hash == source_hash,
+                )
+            )
+        if not existing:
+            existing = EmployeePublication(employee_id=employee.id, source_hash=source_hash, title=_publication_title(publication))
+            db.add(existing)
+        _apply_publication(existing, publication, source_hash)
+
+    if seen_hashes:
+        stale = db.scalars(
+            select(EmployeePublication).where(
+                EmployeePublication.employee_id == employee.id,
+                EmployeePublication.source_hash.not_in(seen_hashes),
+            )
+        ).all()
+        for item in stale:
+            db.delete(item)
+
+
+def _publication_payloads(parsed: dict) -> list[dict]:
+    publications = []
+    for section in parsed.get("sections") or []:
+        if not isinstance(section, dict) or section.get("type") != "publications":
+            continue
+        for publication in section.get("publications") or []:
+            if isinstance(publication, dict):
+                publications.append(publication)
+    return publications
+
+
+def _apply_publication(target: EmployeePublication, publication: dict, source_hash: str) -> None:
+    target.publication_id = _clean_optional(publication.get("publication_id") or publication.get("id"))
+    target.title = _publication_title(publication)
+    target.year = _int_or_none(publication.get("year"))
+    target.publication_type = _clean_optional(publication.get("publication_type") or publication.get("type"))
+    target.language = _clean_optional(publication.get("language"))
+    target.status = _int_or_none(publication.get("status"))
+    target.url = _clean_optional(publication.get("url"))
+    target.doi_url = _clean_optional(publication.get("doi_url"))
+    target.other_url = _clean_optional(publication.get("other_url"))
+    target.document_url = _clean_optional(publication.get("document_url"))
+    target.citation_text = _clean_optional(publication.get("citation_text") or publication.get("text"))
+    target.annotation = publication.get("annotation") if isinstance(publication.get("annotation"), dict) else None
+    target.description = publication.get("description") if isinstance(publication.get("description"), dict) else None
+    target.authors = publication.get("authors") if isinstance(publication.get("authors"), list) else None
+    target.raw_data = publication.get("raw_data") if isinstance(publication.get("raw_data"), dict) else publication
+    target.source_hash = source_hash
+
+
+def _publication_hash(publication: dict) -> str:
+    return _payload_hash(publication.get("raw_data") if isinstance(publication.get("raw_data"), dict) else publication)
+
+
+def _payload_hash(value: object) -> str:
+    payload = json.dumps(_stable_checksum_payload(value), ensure_ascii=False, sort_keys=True, separators=(",", ":"), default=str)
+    return hashlib.sha256(payload.encode("utf-8")).hexdigest()
+
+
+def _publication_title(publication: dict) -> str:
+    return _clean_optional(publication.get("title") or publication.get("text") or publication.get("id")) or "Untitled publication"
+
+
+def _clean_optional(value: object) -> str | None:
+    text = str(value or "").strip()
+    return text or None
+
+
+def _int_or_none(value: object) -> int | None:
+    try:
+        return int(value)
+    except (TypeError, ValueError):
+        return None
+
+
+def _mark_dismissed(db: Session, run: CrawlRun, found_keys: set[str], session: requests.Session, timeout: int) -> int:
     dismissed = 0
     active = db.scalars(select(Employee).where(Employee.status == "active")).all()
     now = datetime.now(timezone.utc)
     for employee in active:
         if employee.profile_key in found_keys:
             continue
+        profile_available = _profile_is_available(session, employee.canonical_url, timeout)
+        if profile_available:
+            _record_employee_change(
+                db,
+                run,
+                employee,
+                "missing_from_source",
+                profile_available=True,
+                message="Профиль доступен, но ссылка отсутствует в исходном списке.",
+            )
+            continue
         employee.status = "dismissed"
         employee.dismissed_at = now
+        _record_employee_change(
+            db,
+            run,
+            employee,
+            "dismissed",
+            profile_available=False,
+            message="Сотрудник отсутствует в исходном списке, профиль не подтвердился как доступный.",
+        )
         dismissed += 1
     db.commit()
     return dismissed
 
 
+def _profile_is_available(session: requests.Session, url: str, timeout: int) -> bool:
+    try:
+        response = session.get(url, headers=HEADERS, timeout=timeout, allow_redirects=True)
+        return response.status_code < 400
+    except requests.RequestException:
+        return False
+
+
+def _record_employee_change(
+    db: Session,
+    run: CrawlRun,
+    employee: Employee,
+    change_type: str,
+    *,
+    profile_available: bool | None,
+    message: str,
+) -> None:
+    db.add(
+        CrawlRunEmployeeChange(
+            crawl_run_id=run.id,
+            employee_id=employee.id,
+            profile_key=employee.profile_key,
+            profile_url=employee.canonical_url,
+            full_name=employee.full_name,
+            change_type=change_type,
+            profile_available=profile_available,
+            message=message,
+        )
+    )
+
+
 def _checksum(data: dict) -> str:
-    payload = json.dumps(data, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
+    payload = json.dumps(_stable_checksum_payload(data), ensure_ascii=False, sort_keys=True, separators=(",", ":"))
     return hashlib.sha256(payload.encode("utf-8")).hexdigest()
+
+
+def _stable_checksum_payload(value):
+    if isinstance(value, dict):
+        return {key: _stable_checksum_payload(item) for key, item in value.items()}
+    if isinstance(value, list):
+        return [_stable_checksum_payload(item) for item in value]
+    if isinstance(value, str):
+        return _normalize_date_dependent_experience(value)
+    return value
+
+
+def _normalize_date_dependent_experience(value: str) -> str:
+    return re.sub(
+        r"(?i)(стаж(?:\s+работы)?(?:\s+в\s+ниу\s+вшэ|\s+в\s+вшэ)?\s*:?\s*)\d+\s*(?:год(?:а|ов)?|лет)",
+        r"\1<experience-years>",
+        value,
+    )

app/services/dataset_versions.py

@@ -0,0 +1,227 @@
+import hashlib
+import json
+from dataclasses import dataclass
+
+from sqlalchemy import desc, select
+from sqlalchemy.orm import Session
+
+from app.models import DatasetVersion, DatasetVersionItem, Employee
+
+
+@dataclass(frozen=True)
+class EmployeeMarker:
+    profile_key: str
+    employee_id: int | None
+    status: str
+    checksum: str
+
+
+def get_or_create_current_version(db: Session, *, crawl_run_id: int | None = None) -> DatasetVersion:
+    employees = db.scalars(select(Employee).order_by(Employee.profile_key)).all()
+    markers = [_employee_marker(employee) for employee in employees]
+    dataset_hash = _dataset_hash(markers)
+    latest = get_latest_version(db)
+    if latest and latest.hash == dataset_hash:
+        return latest
+
+    active_count = sum(1 for marker in markers if marker.status == "active")
+    dismissed_count = sum(1 for marker in markers if marker.status == "dismissed")
+    version = DatasetVersion(
+        hash=dataset_hash,
+        previous_hash=latest.hash if latest else None,
+        crawl_run_id=crawl_run_id,
+        employee_count=len(markers),
+        active_count=active_count,
+        dismissed_count=dismissed_count,
+    )
+    db.add(version)
+    db.flush()
+    for marker in markers:
+        db.add(
+            DatasetVersionItem(
+                dataset_version_id=version.id,
+                profile_key=marker.profile_key,
+                employee_id=marker.employee_id,
+                status=marker.status,
+                checksum=marker.checksum,
+            )
+        )
+    db.flush()
+    return version
+
+
+def get_latest_version(db: Session) -> DatasetVersion | None:
+    return db.scalar(select(DatasetVersion).order_by(desc(DatasetVersion.created_at), desc(DatasetVersion.id)).limit(1))
+
+
+def get_version_by_hash(db: Session, dataset_hash: str | None) -> DatasetVersion | None:
+    if not dataset_hash:
+        return None
+    return db.scalar(select(DatasetVersion).where(DatasetVersion.hash == dataset_hash).limit(1))
+
+
+def service_info_payload(db: Session, *, tools: list[dict], service_name: str, backend_version: str, protocol_version: str) -> dict:
+    version = get_or_create_current_version(db)
+    db.commit()
+    return {
+        "service_name": service_name,
+        "backend_version": backend_version,
+        "protocolVersion": protocol_version,
+        "tools": tools,
+        "dataset": _version_payload(version),
+    }
+
+
+def sync_employees_payload(db: Session, *, client_hash: str | None = None, include_data: bool = True) -> dict:
+    current = get_or_create_current_version(db)
+    db.commit()
+    if not client_hash:
+        return _full_sync_payload(db, current, include_data=include_data, reason=None)
+    if client_hash == current.hash:
+        return {
+            "mode": "delta",
+            "from_hash": client_hash,
+            "to_hash": current.hash,
+            "dataset": _version_payload(current),
+            "changes": {"added": [], "updated": [], "dismissed": [], "removed": []},
+        }
+
+    previous = get_version_by_hash(db, client_hash)
+    if not previous:
+        return _full_sync_payload(db, current, include_data=include_data, reason="unknown_client_hash", from_hash=client_hash)
+
+    return _delta_sync_payload(db, previous, current, include_data=include_data)
+
+
+def _full_sync_payload(
+    db: Session,
+    current: DatasetVersion,
+    *,
+    include_data: bool,
+    reason: str | None,
+    from_hash: str | None = None,
+) -> dict:
+    employees = db.scalars(select(Employee).order_by(Employee.profile_key)).all()
+    payload = {
+        "mode": "full",
+        "from_hash": from_hash,
+        "to_hash": current.hash,
+        "dataset": _version_payload(current),
+        "items": [_employee_payload(employee, include_data=include_data) for employee in employees],
+    }
+    if reason:
+        payload["reason"] = reason
+    return payload
+
+
+def _delta_sync_payload(db: Session, previous: DatasetVersion, current: DatasetVersion, *, include_data: bool) -> dict:
+    previous_items = _items_by_profile_key(previous)
+    current_items = _items_by_profile_key(current)
+    employees = {employee.profile_key: employee for employee in db.scalars(select(Employee)).all()}
+    added = []
+    updated = []
+    dismissed = []
+    removed = []
+
+    for profile_key, current_item in sorted(current_items.items()):
+        previous_item = previous_items.get(profile_key)
+        employee = employees.get(profile_key)
+        if not previous_item:
+            if employee:
+                added.append(_employee_payload(employee, include_data=include_data))
+            continue
+        if previous_item.checksum == current_item.checksum and previous_item.status == current_item.status:
+            continue
+        if current_item.status == "dismissed":
+            dismissed.append(_tombstone(profile_key, current_item.status, employee))
+        elif employee:
+            updated.append(_employee_payload(employee, include_data=include_data))
+
+    for profile_key, previous_item in sorted(previous_items.items()):
+        if profile_key not in current_items:
+            removed.append(_tombstone(profile_key, "removed", employees.get(profile_key), checksum=previous_item.checksum))
+
+    return {
+        "mode": "delta",
+        "from_hash": previous.hash,
+        "to_hash": current.hash,
+        "dataset": _version_payload(current),
+        "changes": {
+            "added": added,
+            "updated": updated,
+            "dismissed": dismissed,
+            "removed": removed,
+        },
+    }
+
+
+def _items_by_profile_key(version: DatasetVersion) -> dict[str, DatasetVersionItem]:
+    return {item.profile_key: item for item in version.items}
+
+
+def _version_payload(version: DatasetVersion) -> dict:
+    return {
+        "hash": version.hash,
+        "previous_hash": version.previous_hash,
+        "created_at": version.created_at.isoformat() if version.created_at else None,
+        "crawl_run_id": version.crawl_run_id,
+        "employee_count": version.employee_count,
+        "active_count": version.active_count,
+        "dismissed_count": version.dismissed_count,
+    }
+
+
+def _employee_marker(employee: Employee) -> EmployeeMarker:
+    return EmployeeMarker(
+        profile_key=employee.profile_key,
+        employee_id=employee.id,
+        status=employee.status,
+        checksum=employee.current_checksum or _payload_hash(employee.current_data or {}),
+    )
+
+
+def _dataset_hash(markers: list[EmployeeMarker]) -> str:
+    payload = [
+        {"profile_key": marker.profile_key, "status": marker.status, "checksum": marker.checksum}
+        for marker in sorted(markers, key=lambda item: item.profile_key)
+    ]
+    return _payload_hash(payload)
+
+
+def _payload_hash(value: object) -> str:
+    payload = json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":"), default=str)
+    return hashlib.sha256(payload.encode("utf-8")).hexdigest()
+
+
+def _employee_payload(employee: Employee, *, include_data: bool) -> dict:
+    payload = {
+        "profile_key": employee.profile_key,
+        "profile_id": employee.profile_id,
+        "full_name": employee.full_name,
+        "status": employee.status,
+        "canonical_url": employee.canonical_url,
+        "last_seen_at": employee.last_seen_at.isoformat() if employee.last_seen_at else None,
+        "dismissed_at": employee.dismissed_at.isoformat() if employee.dismissed_at else None,
+        "checksum": employee.current_checksum or _payload_hash(employee.current_data or {}),
+    }
+    if include_data:
+        payload["data"] = employee.current_data
+    return payload
+
+
+def _tombstone(profile_key: str, status: str, employee: Employee | None, *, checksum: str | None = None) -> dict:
+    payload = {
+        "profile_key": profile_key,
+        "status": status,
+        "checksum": checksum or (employee.current_checksum if employee else None),
+    }
+    if employee:
+        payload.update(
+            {
+                "profile_id": employee.profile_id,
+                "full_name": employee.full_name,
+                "canonical_url": employee.canonical_url,
+                "dismissed_at": employee.dismissed_at.isoformat() if employee.dismissed_at else None,
+            }
+        )
+    return payload

app/services/resource_cache.py

@@ -0,0 +1,147 @@
+from __future__ import annotations
+
+import gzip
+import hashlib
+import json
+from dataclasses import dataclass
+from datetime import datetime, timezone
+from typing import Any
+
+import requests
+from sqlalchemy import select
+from sqlalchemy.orm import Session
+
+from app.models import ParseResourceCache
+from app.version import BACKEND_VERSION
+
+
+@dataclass(frozen=True)
+class CachedResource:
+    text: str
+    body_hash: str
+    from_cache: bool
+    status_code: int
+
+
+class ResourceCache:
+    def __init__(self, db: Session):
+        self.db = db
+
+    def fetch_text(
+        self,
+        session: requests.Session,
+        *,
+        profile_key: str,
+        resource_key: str,
+        method: str,
+        url: str,
+        headers: dict[str, str],
+        timeout: int,
+        json_payload: Any | None = None,
+        params: dict[str, Any] | None = None,
+    ) -> CachedResource:
+        method = method.upper()
+        fingerprint = _request_fingerprint(method=method, url=url, json_payload=json_payload, params=params)
+        cached = self.db.scalar(
+            select(ParseResourceCache).where(
+                ParseResourceCache.profile_key == profile_key,
+                ParseResourceCache.resource_key == resource_key,
+                ParseResourceCache.request_fingerprint == fingerprint,
+            )
+        )
+
+        request_headers = dict(headers)
+        if cached:
+            if cached.etag:
+                request_headers["If-None-Match"] = cached.etag
+            if cached.last_modified:
+                request_headers["If-Modified-Since"] = cached.last_modified
+
+        response = _send(
+            session,
+            method=method,
+            url=url,
+            headers=request_headers,
+            timeout=timeout,
+            json_payload=json_payload,
+            params=params,
+        )
+        if response.status_code == 304 and cached:
+            cached.fetched_at = datetime.now(timezone.utc)
+            self.db.flush()
+            return CachedResource(
+                text=gzip.decompress(cached.body_snapshot).decode("utf-8"),
+                body_hash=cached.body_hash,
+                from_cache=True,
+                status_code=response.status_code,
+            )
+
+        response.raise_for_status()
+        text = response.text
+        body_hash = _body_hash(text)
+        etag = response.headers.get("ETag") if hasattr(response, "headers") else None
+        last_modified = response.headers.get("Last-Modified") if hasattr(response, "headers") else None
+
+        if cached:
+            cached.method = method
+            cached.url = url
+            cached.etag = etag
+            cached.last_modified = last_modified
+            cached.body_hash = body_hash
+            cached.body_snapshot = gzip.compress(text.encode("utf-8"))
+            cached.parser_version = BACKEND_VERSION
+            cached.fetched_at = datetime.now(timezone.utc)
+        else:
+            self.db.add(
+                ParseResourceCache(
+                    profile_key=profile_key,
+                    resource_key=resource_key,
+                    method=method,
+                    url=url,
+                    request_fingerprint=fingerprint,
+                    etag=etag,
+                    last_modified=last_modified,
+                    body_hash=body_hash,
+                    body_snapshot=gzip.compress(text.encode("utf-8")),
+                    parser_version=BACKEND_VERSION,
+                    fetched_at=datetime.now(timezone.utc),
+                )
+            )
+        self.db.flush()
+        return CachedResource(text=text, body_hash=body_hash, from_cache=False, status_code=response.status_code)
+
+
+def _send(
+    session: requests.Session,
+    *,
+    method: str,
+    url: str,
+    headers: dict[str, str],
+    timeout: int,
+    json_payload: Any | None,
+    params: dict[str, Any] | None,
+) -> requests.Response:
+    if method == "POST":
+        return session.post(url, json=json_payload, headers=headers, timeout=timeout, params=params)
+    return session.get(url, headers=headers, timeout=timeout, params=params)
+
+
+def _request_fingerprint(
+    *,
+    method: str,
+    url: str,
+    json_payload: Any | None,
+    params: dict[str, Any] | None,
+) -> str:
+    payload = {
+        "method": method,
+        "url": url,
+        "json": json_payload,
+        "params": params,
+    }
+    encoded = json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
+    return hashlib.sha256(encoded.encode("utf-8")).hexdigest()
+
+
+def _body_hash(text: str) -> str:
+    return hashlib.sha256(text.encode("utf-8")).hexdigest()

app/static/admin.css

@@ -1,6 +1,8 @@
 .admin {
   margin: 0;
   min-height: 100vh;
+  display: flex;
+  flex-direction: column;
   color: #1f2937;
   background: #f6f7f9;
   font-family: Arial, sans-serif;
@@ -21,6 +23,11 @@
   font-size: 20px;
 }
 
+.admin__brand-link {
+  color: inherit;
+  text-decoration: none;
+}
+
 .admin__nav {
   display: flex;
   align-items: center;
@@ -34,6 +41,7 @@
 }
 
 .admin__main {
+  flex: 1;
   width: min(1180px, calc(100% - 32px));
   margin: 28px auto;
 }
@@ -52,18 +60,30 @@
 }
 
 .metric {
+  display: block;
   padding: 18px;
   background: #ffffff;
   border: 1px solid #d9dee7;
   border-radius: 8px;
 }
 
+.metric--link {
+  color: inherit;
+  text-decoration: none;
+}
+
+.metric--link:hover {
+  border-color: #0f766e;
+}
+
 .metric__label {
+  display: block;
   color: #6b7280;
   font-size: 13px;
 }
 
 .metric__value {
+  display: block;
   margin-top: 8px;
   font-size: 28px;
   font-weight: 700;
@@ -87,6 +107,14 @@
   border-collapse: collapse;
 }
 
+.table__row {
+  cursor: pointer;
+}
+
+.table__row:hover {
+  background: #f0fdfa;
+}
+
 .table__cell,
 .table__head {
   padding: 10px 8px;
@@ -143,6 +171,10 @@
   background: transparent;
 }
 
+.button--compact {
+  padding: 8px 12px;
+}
+
 .code {
   overflow-x: auto;
   padding: 14px;
@@ -173,11 +205,34 @@
   gap: 10px;
 }
 
+.employee-card__actions {
+  display: grid;
+  justify-items: end;
+  gap: 10px;
+}
+
 .employee-card__title {
   margin: 0;
   font-size: 24px;
 }
 
+.employee-card__notice {
+  margin: 0;
+  padding: 12px 14px;
+  border-radius: 8px;
+  font-weight: 700;
+}
+
+.employee-card__notice--success {
+  color: #065f46;
+  background: #d1fae5;
+}
+
+.employee-card__notice--error {
+  color: #991b1b;
+  background: #fee2e2;
+}
+
 .employee-card__section {
   padding: 20px;
   background: #ffffff;
@@ -331,12 +386,22 @@
 }
 
 .stats-strip__item {
+  display: block;
   padding: 14px 16px;
   background: #ffffff;
   border: 1px solid #d9dee7;
   border-radius: 8px;
 }
 
+.stats-strip__item--link {
+  color: inherit;
+  text-decoration: none;
+}
+
+.stats-strip__item--link:hover {
+  border-color: #0f766e;
+}
+
 .stats-strip__label {
   display: block;
   color: #6b7280;

app/static/admin.js

@@ -59,10 +59,23 @@
         applyColumns(columns);
       });
     });
+  }
+
+  function setupClickableRows() {
+    const openRow = (row) => {
+      window.location.href = row.dataset.rowHref;
+    };
+
     document.querySelectorAll("[data-row-href]").forEach((row) => {
       row.addEventListener("click", (event) => {
         if (event.target.closest("a, button, input, select, label")) return;
-        window.location.href = row.dataset.rowHref;
+        openRow(row);
+      });
+      row.addEventListener("keydown", (event) => {
+        if (!["Enter", " "].includes(event.key)) return;
+        if (event.target.closest("a, button, input, select, label")) return;
+        event.preventDefault();
+        openRow(row);
       });
     });
   }
@@ -76,12 +89,14 @@
       const status = document.querySelector("[data-progress-status]");
       const processed = document.querySelector("[data-progress-processed]");
       const found = document.querySelector("[data-progress-found]");
+      const skipped = document.querySelector("[data-progress-skipped]");
       const errors = document.querySelector("[data-progress-errors]");
       const fill = document.querySelector("[data-progress-fill]");
       const percent = document.querySelector("[data-progress-percent]");
       if (status) status.textContent = run.status_display || run.status;
       if (processed) processed.textContent = run.processed_count;
       if (found) found.textContent = run.found_count;
+      if (skipped) skipped.textContent = run.skipped_count;
       if (errors) errors.textContent = run.error_count;
       if (fill) fill.style.width = `${run.progress_percent}%`;
       if (percent) percent.textContent = run.progress_percent;
@@ -107,5 +122,6 @@
   }
 
   setupColumns();
+  setupClickableRows();
   setupProgress();
 })();

app/templates/base.html

@@ -8,7 +8,7 @@
   </head>
   <body class="admin">
     <header class="admin__header">
-      <h1 class="admin__brand">MIEM Employees</h1>
+      <h1 class="admin__brand"><a class="admin__brand-link" href="/admin">MIEM Employees</a></h1>
       <nav class="admin__nav">
         <a class="admin__link" href="/admin">Обзор</a>
         <a class="admin__link" href="/admin/directory">Сотрудники</a>

app/templates/dashboard.html

@@ -2,10 +2,10 @@
 {% block title %}Обзор · MIEM Employees{% endblock %}
 {% block content %}
 <section class="admin__grid">
-  <div class="metric"><div class="metric__label">Всего в базе</div><div class="metric__value">{{ counts.total }}</div></div>
+  <a class="metric metric--link" href="/admin/directory"><span class="metric__label">Всего в базе</span><span class="metric__value">{{ counts.total }}</span></a>
-  <div class="metric"><div class="metric__label">Работают</div><div class="metric__value">{{ counts.active }}</div></div>
+  <a class="metric metric--link" href="/admin/directory?status=active"><span class="metric__label">Работают</span><span class="metric__value">{{ counts.active }}</span></a>
-  <div class="metric"><div class="metric__label">Новые за запуск</div><div class="metric__value">{{ counts.new_in_last_run }}</div></div>
+  <a class="metric metric--link" href="{% if latest_run %}/admin/runs/{{ latest_run.id }}#new-employees{% else %}/admin/runs{% endif %}"><span class="metric__label">Новые за запуск</span><span class="metric__value">{{ counts.new_in_last_run }}</span></a>
-  <div class="metric"><div class="metric__label">Уволены</div><div class="metric__value">{{ counts.dismissed }}</div></div>
+  <a class="metric metric--link" href="/admin/directory?status=dismissed"><span class="metric__label">Уволены</span><span class="metric__value">{{ counts.dismissed }}</span></a>
 </section>
 <section class="stats-strip">
   <div class="stats-strip__item">
@@ -16,10 +16,10 @@
     <span class="stats-strip__value">Сотрудников пока нет</span>
     {% endif %}
   </div>
-  <div class="stats-strip__item">
+  <a class="stats-strip__item stats-strip__item--link" href="/admin/runs">
     <span class="stats-strip__label">Запуски</span>
     <span class="stats-strip__value">{{ counts.runs }}</span>
-  </div>
+  </a>
   <div class="stats-strip__item">
     <span class="stats-strip__label">Ошибки</span>
     <span class="stats-strip__value">{{ counts.errors }}</span>
@@ -37,6 +37,7 @@
     <div class="progress-panel__meta">
       <span data-progress-status>{{ run.status_display if run else "Ожидание" }}</span>
       <span>обработано: <span data-progress-processed>{{ run.processed_count if run else 0 }}</span> / <span data-progress-found>{{ run.found_count if run else 0 }}</span></span>
+      <span>без изменений: <span data-progress-skipped>{{ run.skipped_count if run else 0 }}</span></span>
       <span>ошибок: <span data-progress-errors>{{ run.error_count if run else 0 }}</span></span>
     </div>
     <div class="progress-bar" aria-label="Parsing progress">
@@ -48,10 +49,10 @@
 <section class="panel">
   <h2 class="panel__title">Последние запуски</h2>
   <table class="table">
-    <thead><tr><th class="table__head">ID</th><th class="table__head">Статус</th><th class="table__head">Обработано</th><th class="table__head">Ошибки</th><th class="table__head">Старт</th></tr></thead>
+    <thead><tr><th class="table__head">ID</th><th class="table__head">Статус</th><th class="table__head">Обработано</th><th class="table__head">Без изменений</th><th class="table__head">Ошибки</th><th class="table__head">Старт</th></tr></thead>
     <tbody>
       {% for run in runs %}
-      <tr><td class="table__cell">{{ run.id }}</td><td class="table__cell">{{ run.status_display }}</td><td class="table__cell">{{ run.parsed_count }}</td><td class="table__cell">{{ run.error_count }}</td><td class="table__cell">{{ run.started_display }}</td></tr>
+      <tr class="table__row" onclick="window.location.href='/admin/runs/{{ run.id }}'" onkeydown="if (event.key === 'Enter' || event.key === ' ') { event.preventDefault(); window.location.href='/admin/runs/{{ run.id }}'; }" role="link" tabindex="0"><td class="table__cell">{{ run.id }}</td><td class="table__cell">{{ run.status_display }}</td><td class="table__cell">{{ run.parsed_count }}</td><td class="table__cell">{{ run.skipped_count }}</td><td class="table__cell">{{ run.error_count }}</td><td class="table__cell">{{ run.started_display }}</td></tr>
       {% endfor %}
     </tbody>
   </table>

app/templates/employee_detail.html

@@ -7,8 +7,18 @@
       <h2 class="employee-card__title">{{ employee_view.full_name or employee.profile_key }}</h2>
       <span class="badge {% if employee_view.status == "dismissed" %}badge--dismissed{% endif %}">{{ employee_view.status_display }}</span>
     </div>
-    <a class="admin__link" href="{{ employee_view.canonical_url }}">{{ employee_view.canonical_url }}</a>
+    <div class="employee-card__actions">
+      <form method="post" action="/admin/employees/{{ employee.id }}/refresh">
+        <button class="button button--compact" type="submit">Обновить данные</button>
+      </form>
+      <a class="admin__link" href="{{ employee_view.canonical_url }}">{{ employee_view.canonical_url }}</a>
+    </div>
   </div>
+  {% if refresh_status == "success" %}
+  <p class="employee-card__notice employee-card__notice--success">Данные сотрудника обновлены.</p>
+  {% elif refresh_status == "error" %}
+  <p class="employee-card__notice employee-card__notice--error">Не удалось обновить данные сотрудника.</p>
+  {% endif %}
 
   <section class="employee-card__section">
     <h3 class="employee-section__title">Основная информация</h3>

app/templates/run_detail.html

@@ -0,0 +1,65 @@
+{% extends "base.html" %}
+{% block title %}Запуск {{ run.id }} · MIEM Employees{% endblock %}
+{% block content %}
+<section class="panel">
+  <div class="progress-panel__header">
+    <div>
+      <h2 class="panel__title">Запуск {{ run.id }}</h2>
+      <p class="progress-panel__empty">{{ run.started_display }} · {{ run.status_display }}</p>
+    </div>
+    <a class="admin__link" href="/admin/runs">Все запуски</a>
+  </div>
+  <div class="stats-strip">
+    <div class="stats-strip__item"><span class="stats-strip__label">Найдено</span><span class="stats-strip__value">{{ run.found_count }}</span></div>
+    <div class="stats-strip__item"><span class="stats-strip__label">Обработано</span><span class="stats-strip__value">{{ run.parsed_count }}</span></div>
+    <div class="stats-strip__item"><span class="stats-strip__label">Без изменений</span><span class="stats-strip__value">{{ run.skipped_count }}</span></div>
+    <div class="stats-strip__item"><span class="stats-strip__label">Новые</span><span class="stats-strip__value">{{ run.new_count }}</span></div>
+    <div class="stats-strip__item"><span class="stats-strip__label">Потеряшки</span><span class="stats-strip__value">{{ run.changes.missing_from_source | length }}</span></div>
+    <div class="stats-strip__item"><span class="stats-strip__label">Уволены</span><span class="stats-strip__value">{{ run.dismissed_count }}</span></div>
+    <div class="stats-strip__item"><span class="stats-strip__label">Ошибки</span><span class="stats-strip__value">{{ run.error_count }}</span></div>
+  </div>
+  {% if not run.changes_detail_available %}
+  <p class="progress-panel__empty">Детализация сотрудников для этого запуска недоступна. Она сохраняется только для новых запусков после обновления.</p>
+  {% endif %}
+</section>
+
+{% for group, title in [("new", "Новые сотрудники"), ("missing_from_source", "Потеряшки"), ("dismissed", "Уволенные")] %}
+<section class="panel"{% if group == "new" %} id="new-employees"{% endif %}>
+  <h2 class="panel__title">{{ title }}</h2>
+  {% set items = run.changes[group] %}
+  {% if items %}
+  <table class="table">
+    <thead><tr><th class="table__head">ФИО</th><th class="table__head">Профиль</th><th class="table__head">Проверка</th><th class="table__head">Комментарий</th></tr></thead>
+    <tbody>
+      {% for item in items %}
+      <tr>
+        <td class="table__cell">{% if item.employee_id %}<a class="admin__link" href="/admin/employees/{{ item.employee_id }}">{{ item.full_name or item.profile_key }}</a>{% else %}{{ item.full_name or item.profile_key }}{% endif %}</td>
+        <td class="table__cell"><a class="admin__link" href="{{ item.profile_url }}">{{ item.profile_url }}</a></td>
+        <td class="table__cell">{{ item.profile_available_display }}</td>
+        <td class="table__cell">{{ item.message or "" }}</td>
+      </tr>
+      {% endfor %}
+    </tbody>
+  </table>
+  {% else %}
+  <p class="progress-panel__empty">Нет записей.</p>
+  {% endif %}
+</section>
+{% endfor %}
+
+<section class="panel">
+  <h2 class="panel__title">Ошибки запуска</h2>
+  {% if run.errors %}
+  <table class="table">
+    <thead><tr><th class="table__head">Профиль</th><th class="table__head">Ошибка</th><th class="table__head">Время</th></tr></thead>
+    <tbody>
+      {% for error in run.errors %}
+      <tr><td class="table__cell">{{ error.profile_url or "" }}</td><td class="table__cell">{{ error.error_type }}: {{ error.message }}</td><td class="table__cell">{{ error.created_display }}</td></tr>
+      {% endfor %}
+    </tbody>
+  </table>
+  {% else %}
+  <p class="progress-panel__empty">Ошибок нет.</p>
+  {% endif %}
+</section>
+{% endblock %}

app/templates/runs.html

@@ -8,12 +8,13 @@
   </div>
   {% set run = runs[0] if runs else none %}
   {% if run %}
-  {% set processed = run.parsed_count + run.error_count %}
+  {% set processed = run.parsed_count + run.skipped_count + run.error_count %}
   {% set percent = ((processed / run.found_count) * 100) | round(1) if run.found_count else 0 %}
   <div class="progress-panel" data-progress-panel>
     <div class="progress-panel__meta">
       <span data-progress-status>{{ run.status_display }}</span>
       <span>обработано: <span data-progress-processed>{{ processed }}</span> / <span data-progress-found>{{ run.found_count }}</span></span>
+      <span>без изменений: <span data-progress-skipped>{{ run.skipped_count }}</span></span>
       <span>ошибок: <span data-progress-errors>{{ run.error_count }}</span></span>
     </div>
     <div class="progress-bar" aria-label="Parsing progress">
@@ -26,6 +27,7 @@
     <div class="progress-panel__meta">
       <span data-progress-status>Ожидание</span>
       <span>обработано: <span data-progress-processed>0</span> / <span data-progress-found>0</span></span>
+      <span>без изменений: <span data-progress-skipped>0</span></span>
       <span>ошибок: <span data-progress-errors>0</span></span>
     </div>
     <div class="progress-bar" aria-label="Parsing progress">
@@ -35,10 +37,10 @@
   </div>
   {% endif %}
   <table class="table">
-    <thead><tr><th class="table__head">ID</th><th class="table__head">Статус</th><th class="table__head">Найдено</th><th class="table__head">Обработано</th><th class="table__head">Новые</th><th class="table__head">Ошибки</th><th class="table__head">Уволены</th><th class="table__head">Старт</th></tr></thead>
+    <thead><tr><th class="table__head">ID</th><th class="table__head">Статус</th><th class="table__head">Найдено</th><th class="table__head">Обработано</th><th class="table__head">Без изменений</th><th class="table__head">Новые</th><th class="table__head">Ошибки</th><th class="table__head">Уволены</th><th class="table__head">Старт</th></tr></thead>
     <tbody>
       {% for run in runs %}
-      <tr><td class="table__cell">{{ run.id }}</td><td class="table__cell">{{ run.status_display }}</td><td class="table__cell">{{ run.found_count }}</td><td class="table__cell">{{ run.parsed_count }}</td><td class="table__cell">{{ run.new_count }}</td><td class="table__cell">{{ run.error_count }}</td><td class="table__cell">{{ run.dismissed_count }}</td><td class="table__cell">{{ run.started_display }}</td></tr>
+      <tr class="table__row" onclick="window.location.href='/admin/runs/{{ run.id }}'" onkeydown="if (event.key === 'Enter' || event.key === ' ') { event.preventDefault(); window.location.href='/admin/runs/{{ run.id }}'; }" role="link" tabindex="0"><td class="table__cell">{{ run.id }}</td><td class="table__cell">{{ run.status_display }}</td><td class="table__cell">{{ run.found_count }}</td><td class="table__cell">{{ run.parsed_count }}</td><td class="table__cell">{{ run.skipped_count }}</td><td class="table__cell">{{ run.new_count }}</td><td class="table__cell">{{ run.error_count }}</td><td class="table__cell">{{ run.dismissed_count }}</td><td class="table__cell">{{ run.started_display }}</td></tr>
       {% endfor %}
     </tbody>
   </table>

app/version.py

@@ -1,3 +1,3 @@
-APP_VERSION = "0.3.0"
+APP_VERSION = "0.6.2"
-FRONTEND_VERSION = "0.3.0"
+FRONTEND_VERSION = "0.6.2"
-BACKEND_VERSION = "0.3.0"
+BACKEND_VERSION = "0.6.2"

app/worker.py

@@ -17,7 +17,14 @@ def crawl_once() -> None:
     settings = get_settings()
     with SessionLocal() as db:
         run = run_crawl(db, settings)
-        logger.info("crawl finished: id=%s status=%s parsed=%s errors=%s", run.id, run.status, run.parsed_count, run.error_count)
+        logger.info(
+            "crawl finished: id=%s status=%s parsed=%s skipped=%s errors=%s",
+            run.id,
+            run.status,
+            run.parsed_count,
+            run.skipped_count,
+            run.error_count,
+        )
 
 
 def main() -> None:

docker-compose.yml

@@ -20,7 +20,7 @@ services:
     environment:
       DATABASE_URL: postgresql+psycopg://${POSTGRES_USER:-miem}:${POSTGRES_PASSWORD:-miem_password}@postgres:5432/${POSTGRES_DB:-miem_workers}
     ports:
-      - "127.0.0.1:8000:8000"
+      - "127.0.0.1:${API_PORT:-8000}:8000"
     depends_on:
       postgres:
         condition: service_healthy
@@ -42,7 +42,7 @@ services:
     environment:
       DATABASE_URL: postgresql+psycopg://${POSTGRES_USER:-miem}:${POSTGRES_PASSWORD:-miem_password}@postgres:5432/${POSTGRES_DB:-miem_workers}
     ports:
-      - "127.0.0.1:8001:8000"
+      - "127.0.0.1:${MCP_PORT:-8001}:8000"
     depends_on:
       postgres:
         condition: service_healthy

migrations/001_init.sql

@@ -13,6 +13,7 @@ CREATE TABLE IF NOT EXISTS crawl_runs (
   finished_at TIMESTAMPTZ,
   found_count INTEGER NOT NULL DEFAULT 0,
   parsed_count INTEGER NOT NULL DEFAULT 0,
+  skipped_count INTEGER NOT NULL DEFAULT 0,
   new_count INTEGER NOT NULL DEFAULT 0,
   error_count INTEGER NOT NULL DEFAULT 0,
   dismissed_count INTEGER NOT NULL DEFAULT 0,
@@ -73,3 +74,22 @@ CREATE TABLE IF NOT EXISTS profile_tabs (
 );
 
 CREATE INDEX IF NOT EXISTS ix_profile_tabs_employee_id ON profile_tabs (employee_id);
+
+CREATE TABLE IF NOT EXISTS parse_resource_cache (
+  id SERIAL PRIMARY KEY,
+  profile_key VARCHAR(255) NOT NULL,
+  resource_key VARCHAR(255) NOT NULL,
+  method VARCHAR(16) NOT NULL,
+  url TEXT NOT NULL,
+  request_fingerprint VARCHAR(64) NOT NULL,
+  etag TEXT,
+  last_modified TEXT,
+  body_hash VARCHAR(64) NOT NULL,
+  body_snapshot BYTEA NOT NULL,
+  parser_version VARCHAR(32),
+  fetched_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+  CONSTRAINT uq_parse_resource_cache_resource UNIQUE (profile_key, resource_key, request_fingerprint)
+);
+
+CREATE INDEX IF NOT EXISTS ix_parse_resource_cache_profile_key
+  ON parse_resource_cache (profile_key);

migrations/003_crawl_run_employee_changes.sql

@@ -0,0 +1,21 @@
+CREATE TABLE IF NOT EXISTS crawl_run_employee_changes (
+  id SERIAL PRIMARY KEY,
+  crawl_run_id INTEGER NOT NULL REFERENCES crawl_runs(id),
+  employee_id INTEGER REFERENCES employees(id),
+  profile_key VARCHAR(255) NOT NULL,
+  profile_url TEXT NOT NULL,
+  full_name TEXT,
+  change_type VARCHAR(32) NOT NULL,
+  profile_available BOOLEAN,
+  message TEXT,
+  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+CREATE INDEX IF NOT EXISTS ix_crawl_run_employee_changes_run_id
+  ON crawl_run_employee_changes (crawl_run_id);
+
+CREATE INDEX IF NOT EXISTS ix_crawl_run_employee_changes_employee_id
+  ON crawl_run_employee_changes (employee_id);
+
+CREATE INDEX IF NOT EXISTS ix_crawl_run_employee_changes_change_type
+  ON crawl_run_employee_changes (change_type);

migrations/004_dataset_versions.sql

@@ -0,0 +1,29 @@
+CREATE TABLE IF NOT EXISTS dataset_versions (
+  id SERIAL PRIMARY KEY,
+  hash VARCHAR(64) NOT NULL UNIQUE,
+  previous_hash VARCHAR(64),
+  crawl_run_id INTEGER REFERENCES crawl_runs(id),
+  employee_count INTEGER NOT NULL DEFAULT 0,
+  active_count INTEGER NOT NULL DEFAULT 0,
+  dismissed_count INTEGER NOT NULL DEFAULT 0,
+  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+CREATE INDEX IF NOT EXISTS ix_dataset_versions_created_at
+  ON dataset_versions (created_at);
+
+CREATE TABLE IF NOT EXISTS dataset_version_items (
+  id SERIAL PRIMARY KEY,
+  dataset_version_id INTEGER NOT NULL REFERENCES dataset_versions(id),
+  profile_key VARCHAR(255) NOT NULL,
+  employee_id INTEGER REFERENCES employees(id),
+  status VARCHAR(32) NOT NULL,
+  checksum VARCHAR(64) NOT NULL,
+  CONSTRAINT uq_dataset_version_items_version_profile UNIQUE (dataset_version_id, profile_key)
+);
+
+CREATE INDEX IF NOT EXISTS ix_dataset_version_items_hash
+  ON dataset_version_items (dataset_version_id);
+
+CREATE INDEX IF NOT EXISTS ix_dataset_version_items_profile_key
+  ON dataset_version_items (profile_key);

migrations/005_parse_resource_cache.sql

@@ -0,0 +1,21 @@
+ALTER TABLE crawl_runs
+ADD COLUMN IF NOT EXISTS skipped_count INTEGER NOT NULL DEFAULT 0;
+
+CREATE TABLE IF NOT EXISTS parse_resource_cache (
+  id SERIAL PRIMARY KEY,
+  profile_key VARCHAR(255) NOT NULL,
+  resource_key VARCHAR(255) NOT NULL,
+  method VARCHAR(16) NOT NULL,
+  url TEXT NOT NULL,
+  request_fingerprint VARCHAR(64) NOT NULL,
+  etag TEXT,
+  last_modified TEXT,
+  body_hash VARCHAR(64) NOT NULL,
+  body_snapshot BYTEA NOT NULL,
+  parser_version VARCHAR(32),
+  fetched_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+  CONSTRAINT uq_parse_resource_cache_resource UNIQUE (profile_key, resource_key, request_fingerprint)
+);
+
+CREATE INDEX IF NOT EXISTS ix_parse_resource_cache_profile_key
+  ON parse_resource_cache (profile_key);

migrations/006_employee_publications.sql

@@ -0,0 +1,39 @@
+CREATE TABLE IF NOT EXISTS employee_publications (
+  id SERIAL PRIMARY KEY,
+  employee_id INTEGER NOT NULL REFERENCES employees(id) ON DELETE CASCADE,
+  publication_id VARCHAR(64),
+  title TEXT NOT NULL,
+  year INTEGER,
+  publication_type VARCHAR(64),
+  language VARCHAR(16),
+  status INTEGER,
+  url TEXT,
+  doi_url TEXT,
+  other_url TEXT,
+  document_url TEXT,
+  citation_text TEXT,
+  annotation JSONB,
+  description JSONB,
+  authors JSONB,
+  raw_data JSONB,
+  source_hash VARCHAR(64) NOT NULL,
+  created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+  updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+  CONSTRAINT uq_employee_publications_employee_publication UNIQUE (employee_id, publication_id),
+  CONSTRAINT uq_employee_publications_employee_source_hash UNIQUE (employee_id, source_hash)
+);
+
+CREATE INDEX IF NOT EXISTS ix_employee_publications_employee_id
+  ON employee_publications (employee_id);
+
+CREATE INDEX IF NOT EXISTS ix_employee_publications_publication_id
+  ON employee_publications (publication_id);
+
+CREATE INDEX IF NOT EXISTS ix_employee_publications_doi_url
+  ON employee_publications (doi_url);
+
+CREATE INDEX IF NOT EXISTS ix_employee_publications_year
+  ON employee_publications (year);
+
+CREATE INDEX IF NOT EXISTS ix_employee_publications_publication_type
+  ON employee_publications (publication_type);

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "miem-workers"
-version = "0.3.0"
+version = "0.6.2"
 description = "MIEM employees parser, admin API, and MCP server"
 requires-python = ">=3.11"
 dependencies = [
@@ -12,7 +12,6 @@ dependencies = [
   "lxml>=5.2.0",
   "psycopg[binary]>=3.2.0",
   "pydantic-settings>=2.4.0",
-  "PyJWT[crypto]>=2.9.0",
   "python-multipart>=0.0.9",
   "requests>=2.32.0",
   "sqlalchemy>=2.0.32",

requirements.txt

@@ -6,7 +6,6 @@ jinja2>=3.1.4
 lxml>=5.2.0
 psycopg[binary]>=3.2.0
 pydantic-settings>=2.4.0
-PyJWT[crypto]>=2.9.0
 python-multipart>=0.0.9
 requests>=2.32.0
 sqlalchemy>=2.0.32

tests/test_admin_data.py

@@ -1,11 +1,12 @@
 from datetime import datetime, timezone
 
-from app.models import CrawlRun, Employee
+from app.models import CrawlError, CrawlRun, CrawlRunEmployeeChange, Employee
 from app.services.admin_data import (
     employee_detail_payload,
     employee_display_payload,
     format_admin_datetime,
     list_employees_page,
+    run_detail_payload,
     run_payload,
     stats_payload,
 )
@@ -199,11 +200,52 @@ def test_run_payload_calculates_progress():
         status="running",
         found_count=10,
         parsed_count=4,
+        skipped_count=2,
         error_count=1,
     )
 
     payload = run_payload(run)
 
-    assert payload["processed_count"] == 5
+    assert payload["processed_count"] == 7
-    assert payload["progress_percent"] == 50.0
+    assert payload["progress_percent"] == 70.0
     assert payload["status_display"] == "Выполняется"
+
+
+def test_run_detail_payload_groups_changes_and_handles_old_runs(db_session):
+    old_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="completed")
+    run = CrawlRun(source_url="https://miem.hse.ru/persons", status="completed", new_count=1)
+    employee = Employee(
+        profile_key="staff:new",
+        canonical_url="https://www.hse.ru/staff/new",
+        full_name="New Person",
+        status="active",
+        first_seen_at=datetime.now(timezone.utc),
+        last_seen_at=datetime.now(timezone.utc),
+    )
+    db_session.add_all([old_run, run, employee])
+    db_session.commit()
+    db_session.add(
+        CrawlRunEmployeeChange(
+            crawl_run_id=run.id,
+            employee_id=employee.id,
+            profile_key=employee.profile_key,
+            profile_url=employee.canonical_url,
+            full_name=employee.full_name,
+            change_type="new",
+            profile_available=True,
+            message="added",
+        )
+    )
+    db_session.add(
+        CrawlError(crawl_run_id=run.id, profile_url=employee.canonical_url, error_type="ValueError", message="bad")
+    )
+    db_session.commit()
+
+    payload = run_detail_payload(db_session, run)
+    old_payload = run_detail_payload(db_session, old_run)
+
+    assert payload["changes_detail_available"] is True
+    assert payload["changes"]["new"][0]["full_name"] == "New Person"
+    assert payload["errors"][0]["error_type"] == "ValueError"
+    assert old_payload["changes_detail_available"] is False
+    assert old_payload["changes"]["new"] == []

tests/test_admin_templates.py

@@ -8,6 +8,7 @@ def test_base_navigation_is_russian_and_has_no_legacy_employees_link():
     assert "Сотрудники" in template
     assert "Запуски" in template
     assert "Выйти" in template
+    assert '<a class="admin__brand-link" href="/admin">MIEM Employees</a>' in template
     assert ">Employees<" not in template
     assert "/admin/employees" not in template
 
@@ -32,3 +33,61 @@ def test_admin_employees_route_redirects_to_directory():
     source = Path("app/admin.py").read_text(encoding="utf-8")
 
     assert 'RedirectResponse("/admin/directory", status_code=303)' in source
+
+
+def test_dashboard_limits_latest_runs_to_five():
+    source = Path("app/admin.py").read_text(encoding="utf-8")
+
+    assert "order_by(desc(CrawlRun.started_at)).limit(5)" in source
+    assert "order_by(desc(CrawlRun.started_at)).limit(10)" not in source
+
+
+def test_runs_template_links_to_run_detail():
+    template = Path("app/templates/runs.html").read_text(encoding="utf-8")
+
+    assert 'onclick="window.location.href=\'/admin/runs/{{ run.id }}\'"' in template
+    assert "onkeydown=\"if (event.key === 'Enter' || event.key === ' ')" in template
+    assert 'role="link"' in template
+    assert 'tabindex="0"' in template
+    assert 'data-row-href="/admin/runs/{{ run.id }}"' not in template
+    assert '<a class="admin__link" href="/admin/runs/{{ run.id }}">' not in template
+
+
+def test_run_detail_template_extends_base_and_shows_change_groups():
+    template = Path("app/templates/run_detail.html").read_text(encoding="utf-8")
+
+    assert '{% extends "base.html" %}' in template
+    assert 'id="new-employees"' in template
+    assert "Новые сотрудники" in template
+    assert "Потеряшки" in template
+    assert "Уволенные" in template
+    assert "Детализация сотрудников для этого запуска недоступна" in template
+
+
+def test_dashboard_metric_cards_link_to_admin_targets():
+    template = Path("app/templates/dashboard.html").read_text(encoding="utf-8")
+
+    assert 'href="/admin/directory"' in template
+    assert 'href="/admin/directory?status=active"' in template
+    assert '/admin/runs/{{ latest_run.id }}#new-employees' in template
+    assert 'href="/admin/directory?status=dismissed"' in template
+    assert 'href="/admin/runs"' in template
+
+
+def test_dashboard_latest_run_rows_link_to_run_detail():
+    template = Path("app/templates/dashboard.html").read_text(encoding="utf-8")
+
+    assert 'onclick="window.location.href=\'/admin/runs/{{ run.id }}\'"' in template
+    assert "onkeydown=\"if (event.key === 'Enter' || event.key === ' ')" in template
+    assert 'role="link"' in template
+    assert 'tabindex="0"' in template
+    assert 'data-row-href="/admin/runs/{{ run.id }}"' not in template
+    assert '<a class="admin__link" href="/admin/runs/{{ run.id }}">' not in template
+
+
+def test_admin_js_supports_keyboard_activation_for_clickable_rows():
+    source = Path("app/static/admin.js").read_text(encoding="utf-8")
+
+    assert 'addEventListener("keydown"' in source
+    assert '"Enter"' in source
+    assert '" "' in source

tests/test_api_mcp.py

@@ -1,19 +1,16 @@
-import time
+import json
 from datetime import datetime, timezone
 from types import SimpleNamespace
 
-import jwt
 from fastapi.testclient import TestClient
-from cryptography.hazmat.primitives.asymmetric import rsa
+from sqlalchemy import create_engine, select
-from sqlalchemy import create_engine
 from sqlalchemy.orm import sessionmaker
 from sqlalchemy.pool import StaticPool
 
-import app.security as security
 from app.config import Settings, get_settings
 from app.db import Base, get_db
 from app.main import app
-from app.models import CrawlRun, Employee
+from app.models import CrawlRun, CrawlRunEmployeeChange, Employee, EmployeePublication
 from app.security import SESSION_COOKIE, sign_session
 
 
@@ -23,10 +20,10 @@ def test_health_returns_versions():
     response = client.get("/api/health")
 
     assert response.status_code == 200
-    assert response.json()["backend_version"] == "0.3.0"
+    assert response.json()["backend_version"] == "0.6.2"
 
 
-def test_mcp_requires_token_and_lists_tools():
+def test_mcp_lists_tools_without_auth_and_ignores_auth_header():
     engine = create_engine(
         "sqlite:///:memory:",
         connect_args={"check_same_thread": False},
@@ -43,21 +40,23 @@ def test_mcp_requires_token_and_lists_tools():
             session.close()
 
     app.dependency_overrides[get_db] = override_db
-    app.dependency_overrides[get_settings] = lambda: Settings(
-        mcp_auth_mode="token", mcp_token="secret", session_secret="session-secret"
-    )
     client = TestClient(app)
 
-    unauthorized = client.post("/mcp", json={"jsonrpc": "2.0", "id": 1, "method": "tools/list", "params": {}})
+    without_auth = client.post("/mcp", json={"jsonrpc": "2.0", "id": 1, "method": "tools/list", "params": {}})
-    authorized = client.post(
+    with_auth = client.post(
         "/mcp",
-        headers={"Authorization": "Bearer secret"},
+        headers={"Authorization": "Bearer anything"},
         json={"jsonrpc": "2.0", "id": 1, "method": "tools/list", "params": {}},
     )
 
-    assert unauthorized.status_code == 401
+    assert without_auth.status_code == 200
-    assert authorized.status_code == 200
+    assert with_auth.status_code == 200
-    assert authorized.json()["result"]["tools"][0]["name"] == "search_employees"
+    tool_names = {tool["name"] for tool in without_auth.json()["result"]["tools"]}
+    assert "search_employees" in tool_names
+    assert "get_service_info" in tool_names
+    assert "sync_employees" in tool_names
+    assert any(tool["name"] == "get_crawl_run_details" for tool in without_auth.json()["result"]["tools"])
+    assert with_auth.json()["result"]["tools"] == without_auth.json()["result"]["tools"]
 
     app.dependency_overrides.clear()
 
@@ -95,14 +94,10 @@ def test_mcp_search_employees_returns_matching_employee():
             db.close()
 
     app.dependency_overrides[get_db] = override_db
-    app.dependency_overrides[get_settings] = lambda: Settings(
-        mcp_auth_mode="token", mcp_token="secret", session_secret="session-secret"
-    )
     client = TestClient(app)
 
     response = client.post(
         "/mcp",
-        headers={"Authorization": "Bearer secret"},
         json={
             "jsonrpc": "2.0",
             "id": 1,
@@ -117,7 +112,7 @@ def test_mcp_search_employees_returns_matching_employee():
     app.dependency_overrides.clear()
 
 
-def test_mcp_oauth_rejects_static_token():
+def test_mcp_service_info_returns_tools_and_dataset_hash():
     engine = create_engine(
         "sqlite:///:memory:",
         connect_args={"check_same_thread": False},
@@ -125,138 +120,294 @@ def test_mcp_oauth_rejects_static_token():
     )
     Base.metadata.create_all(engine)
     Session = sessionmaker(bind=engine)
+    session = Session()
+    session.add(
+        Employee(
+            profile_key="staff:alpha",
+            profile_type="staff",
+            profile_id="alpha",
+            canonical_url="https://www.hse.ru/staff/alpha",
+            full_name="Alpha Person",
+            status="active",
+            current_checksum="a" * 64,
+            current_data={"sections": []},
+        )
+    )
+    session.commit()
+    session.close()
 
     def override_db():
-        session = Session()
+        db = Session()
         try:
-            yield session
+            yield db
         finally:
-            session.close()
+            db.close()
 
-    settings = Settings(
-        mcp_auth_mode="oauth",
-        mcp_token="secret",
-        session_secret="session-secret",
-        mcp_oauth_issuer="https://auth.example.com",
-        mcp_oauth_audience="miem-mcp",
-        mcp_oauth_jwks_url="https://auth.example.com/.well-known/jwks.json",
-    )
     app.dependency_overrides[get_db] = override_db
-    app.dependency_overrides[get_settings] = lambda: settings
     client = TestClient(app)
 
     response = client.post(
         "/mcp",
-        headers={"Authorization": "Bearer secret"},
+        json={"jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": {"name": "get_service_info", "arguments": {}}},
-        json={"jsonrpc": "2.0", "id": 1, "method": "tools/list", "params": {}},
-    )
-
-    assert response.status_code == 401
-    assert response.headers["www-authenticate"] == (
-        'Bearer resource_metadata="http://localhost:8001/.well-known/oauth-protected-resource"'
-    )
-
-    app.dependency_overrides.clear()
-
-
-def test_mcp_oauth_missing_auth_returns_metadata_challenge():
-    settings = Settings(
-        mcp_auth_mode="oauth",
-        mcp_resource_url="https://api.example.com/mcp",
-        mcp_oauth_issuer="https://auth.example.com",
-        mcp_oauth_audience="miem-mcp",
-        mcp_oauth_jwks_url="https://auth.example.com/.well-known/jwks.json",
-    )
-    app.dependency_overrides[get_settings] = lambda: settings
-    client = TestClient(app)
-
-    response = client.post("/mcp", json={"jsonrpc": "2.0", "id": 1, "method": "tools/list", "params": {}})
-
-    assert response.status_code == 401
-    assert response.headers["www-authenticate"] == (
-        'Bearer resource_metadata="https://api.example.com/.well-known/oauth-protected-resource"'
-    )
-
-    app.dependency_overrides.clear()
-
-
-def test_mcp_accepts_valid_oauth_jwt(monkeypatch):
-    public_key, token = _oauth_key_and_token()
-    monkeypatch.setattr(security, "_get_mcp_oauth_signing_key", lambda _token, _settings: SimpleNamespace(key=public_key))
-    app.dependency_overrides[get_settings] = lambda: _oauth_settings()
-    client = TestClient(app)
-
-    response = client.post(
-        "/mcp",
-        headers={"Authorization": f"Bearer {token}"},
-        json={"jsonrpc": "2.0", "id": 1, "method": "tools/list", "params": {}},
     )
 
     assert response.status_code == 200
-    assert response.json()["result"]["tools"][0]["name"] == "search_employees"
+    payload = json.loads(response.json()["result"]["content"][0]["text"])
+    assert payload["service_name"] == "miem-employees"
+    assert payload["backend_version"] == "0.6.2"
+    assert payload["dataset"]["hash"]
+    assert any(tool["name"] == "sync_employees" for tool in payload["tools"])
 
     app.dependency_overrides.clear()
 
 
-def test_mcp_rejects_invalid_oauth_jwts(monkeypatch):
+def test_mcp_list_employee_publications_prefers_stored_publications_with_fallback():
-    public_key, expired_token = _oauth_key_and_token(exp=int(time.time()) - 60)
+    engine = create_engine(
-    _, wrong_issuer_token = _oauth_key_and_token(issuer="https://other.example.com")
+        "sqlite:///:memory:",
-    _, wrong_audience_token = _oauth_key_and_token(audience="other-audience")
+        connect_args={"check_same_thread": False},
-    _, bad_signature_token = _oauth_key_and_token(public_key=public_key)
+        poolclass=StaticPool,
-    monkeypatch.setattr(security, "_get_mcp_oauth_signing_key", lambda _token, _settings: SimpleNamespace(key=public_key))
+    )
-    app.dependency_overrides[get_settings] = lambda: _oauth_settings()
+    Base.metadata.create_all(engine)
+    Session = sessionmaker(bind=engine)
+    session = Session()
+    stored_employee = Employee(
+        profile_key="staff:stored",
+        profile_type="staff",
+        profile_id="stored",
+        canonical_url="https://www.hse.ru/staff/stored",
+        full_name="Stored Person",
+        status="active",
+        current_data={
+            "sections": [
+                {
+                    "type": "publications",
+                    "publications": [{"title": "Old JSON Publication", "url": "https://example.test/old"}],
+                }
+            ]
+        },
+    )
+    fallback_employee = Employee(
+        profile_key="staff:fallback",
+        profile_type="staff",
+        profile_id="fallback",
+        canonical_url="https://www.hse.ru/staff/fallback",
+        full_name="Fallback Person",
+        status="active",
+        current_data={
+            "sections": [
+                {
+                    "type": "publications",
+                    "publications": [{"title": "Fallback Publication", "url": "https://example.test/fallback"}],
+                }
+            ]
+        },
+    )
+    session.add_all([stored_employee, fallback_employee])
+    session.commit()
+    session.add(
+        EmployeePublication(
+            employee_id=stored_employee.id,
+            publication_id="pub-1",
+            title="Stored Publication",
+            year=2024,
+            publication_type="ARTICLE",
+            url="https://publications.hse.ru/view/pub-1",
+            doi_url="https://doi.org/10.1/test",
+            citation_text="Stored Citation",
+            annotation={"ru": "Аннотация", "en": "Abstract"},
+            description={"main": "Stored Citation"},
+            authors=[{"id": "1", "title_ru": "Автор", "is_current_employee": True}],
+            source_hash="a" * 64,
+        )
+    )
+    session.commit()
+    session.close()
+
+    def override_db():
+        db = Session()
+        try:
+            yield db
+        finally:
+            db.close()
+
+    app.dependency_overrides[get_db] = override_db
     client = TestClient(app)
 
-    for token in [expired_token, wrong_issuer_token, wrong_audience_token, bad_signature_token]:
+    stored_response = client.post(
-        response = client.post(
+        "/mcp",
-            "/mcp",
+        json={
-            headers={"Authorization": f"Bearer {token}"},
+            "jsonrpc": "2.0",
-            json={"jsonrpc": "2.0", "id": 1, "method": "tools/list", "params": {}},
+            "id": 1,
-        )
+            "method": "tools/call",
+            "params": {"name": "list_employee_publications", "arguments": {"profile_id_or_url": "stored"}},
+        },
+    )
+    fallback_response = client.post(
+        "/mcp",
+        json={
+            "jsonrpc": "2.0",
+            "id": 2,
+            "method": "tools/call",
+            "params": {"name": "list_employee_publications", "arguments": {"profile_id_or_url": "fallback"}},
+        },
+    )
 
-        assert response.status_code == 401
+    stored_payload = json.loads(stored_response.json()["result"]["content"][0]["text"])
+    fallback_payload = json.loads(fallback_response.json()["result"]["content"][0]["text"])
+    assert stored_payload["items"][0]["title"] == "Stored Publication"
+    assert stored_payload["items"][0]["doi_url"] == "https://doi.org/10.1/test"
+    assert stored_payload["items"][0]["annotation"] == {"ru": "Аннотация", "en": "Abstract"}
+    assert stored_payload["items"][0]["authors"] == [{"id": "1", "title_ru": "Автор", "is_current_employee": True}]
+    assert fallback_payload["items"][0]["title"] == "Fallback Publication"
 
     app.dependency_overrides.clear()
 
 
-def test_mcp_rejects_oauth_jwt_without_required_scope(monkeypatch):
+def test_mcp_sync_employees_full_empty_and_unknown_hash_modes():
-    public_key, token = _oauth_key_and_token(scope="profile")
+    engine = create_engine(
-    monkeypatch.setattr(security, "_get_mcp_oauth_signing_key", lambda _token, _settings: SimpleNamespace(key=public_key))
+        "sqlite:///:memory:",
-    app.dependency_overrides[get_settings] = lambda: _oauth_settings()
+        connect_args={"check_same_thread": False},
+        poolclass=StaticPool,
+    )
+    Base.metadata.create_all(engine)
+    Session = sessionmaker(bind=engine)
+    session = Session()
+    session.add(
+        Employee(
+            profile_key="staff:alpha",
+            profile_type="staff",
+            profile_id="alpha",
+            canonical_url="https://www.hse.ru/staff/alpha",
+            full_name="Alpha Person",
+            status="active",
+            current_checksum="a" * 64,
+            current_data={"sections": [{"type": "paragraphs"}]},
+        )
+    )
+    session.commit()
+    session.close()
+
+    def override_db():
+        db = Session()
+        try:
+            yield db
+        finally:
+            db.close()
+
+    app.dependency_overrides[get_db] = override_db
+    client = TestClient(app)
+
+    full_response = client.post(
+        "/mcp",
+        json={"jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": {"name": "sync_employees", "arguments": {}}},
+    )
+    full_payload = json.loads(full_response.json()["result"]["content"][0]["text"])
+    current_hash = full_payload["to_hash"]
+
+    empty_response = client.post(
+        "/mcp",
+        json={
+            "jsonrpc": "2.0",
+            "id": 2,
+            "method": "tools/call",
+            "params": {"name": "sync_employees", "arguments": {"client_hash": current_hash}},
+        },
+    )
+    empty_payload = json.loads(empty_response.json()["result"]["content"][0]["text"])
+
+    unknown_response = client.post(
+        "/mcp",
+        json={
+            "jsonrpc": "2.0",
+            "id": 3,
+            "method": "tools/call",
+            "params": {"name": "sync_employees", "arguments": {"client_hash": "missing"}},
+        },
+    )
+    unknown_payload = json.loads(unknown_response.json()["result"]["content"][0]["text"])
+
+    assert full_payload["mode"] == "full"
+    assert full_payload["items"][0]["data"] == {"sections": [{"type": "paragraphs"}]}
+    assert empty_payload["mode"] == "delta"
+    assert empty_payload["changes"] == {"added": [], "updated": [], "dismissed": [], "removed": []}
+    assert unknown_payload["mode"] == "full"
+    assert unknown_payload["reason"] == "unknown_client_hash"
+
+    app.dependency_overrides.clear()
+
+
+def test_mcp_get_crawl_run_details_returns_changes():
+    engine = create_engine(
+        "sqlite:///:memory:",
+        connect_args={"check_same_thread": False},
+        poolclass=StaticPool,
+    )
+    Base.metadata.create_all(engine)
+    Session = sessionmaker(bind=engine)
+    session = Session()
+    run = CrawlRun(source_url="https://miem.hse.ru/persons", status="completed", new_count=1)
+    employee = Employee(
+        profile_key="staff:new",
+        profile_type="staff",
+        profile_id="new",
+        canonical_url="https://www.hse.ru/staff/new",
+        full_name="New Person",
+        status="active",
+        first_seen_at=datetime.now(timezone.utc),
+        last_seen_at=datetime.now(timezone.utc),
+    )
+    session.add_all([run, employee])
+    session.commit()
+    session.add(
+        CrawlRunEmployeeChange(
+            crawl_run_id=run.id,
+            employee_id=employee.id,
+            profile_key=employee.profile_key,
+            profile_url=employee.canonical_url,
+            full_name=employee.full_name,
+            change_type="new",
+            profile_available=True,
+            message="added",
+        )
+    )
+    session.commit()
+    run_id = run.id
+    session.close()
+
+    def override_db():
+        db = Session()
+        try:
+            yield db
+        finally:
+            db.close()
+
+    app.dependency_overrides[get_db] = override_db
     client = TestClient(app)
 
     response = client.post(
         "/mcp",
-        headers={"Authorization": f"Bearer {token}"},
+        json={
-        json={"jsonrpc": "2.0", "id": 1, "method": "tools/list", "params": {}},
+            "jsonrpc": "2.0",
+            "id": 1,
+            "method": "tools/call",
+            "params": {"name": "get_crawl_run_details", "arguments": {"run_id": run_id}},
+        },
     )
 
-    assert response.status_code == 403
+    assert response.status_code == 200
+    text = response.json()["result"]["content"][0]["text"]
+    assert "New Person" in text
+    assert "changes_detail_available" in text
 
     app.dependency_overrides.clear()
 
 
-def test_mcp_protected_resource_metadata_uses_settings():
+def test_mcp_protected_resource_metadata_route_is_removed():
-    settings = Settings(
-        mcp_resource_url="https://api.example.com/mcp",
-        mcp_oauth_issuer="https://auth.example.com/",
-        mcp_oauth_required_scope="mcp:tools",
-    )
-    app.dependency_overrides[get_settings] = lambda: settings
     client = TestClient(app)
 
     response = client.get("/.well-known/oauth-protected-resource")
 
-    assert response.status_code == 200
+    assert response.status_code == 404
-    assert response.json() == {
-        "resource": "https://api.example.com/mcp",
-        "authorization_servers": ["https://auth.example.com"],
-        "bearer_methods_supported": ["header"],
-        "scopes_supported": ["mcp:tools"],
-        "resource_documentation": "https://api.example.com/mcp",
-    }
-
-    app.dependency_overrides.clear()
 
 
 def test_api_employees_and_stats_require_admin_session():
@@ -281,8 +432,23 @@ def test_api_employees_and_stats_require_admin_session():
             current_data={"contacts": {"emails": ["alpha@hse.ru"]}, "sections": []},
         )
     )
-    db.add(CrawlRun(source_url="https://miem.hse.ru/persons", status="completed", new_count=1))
+    run = CrawlRun(source_url="https://miem.hse.ru/persons", status="completed", new_count=1)
+    db.add(run)
     db.commit()
+    db.add(
+        CrawlRunEmployeeChange(
+            crawl_run_id=run.id,
+            employee_id=1,
+            profile_key="staff:alpha",
+            profile_url="https://www.hse.ru/staff/alpha",
+            full_name="Alpha Person",
+            change_type="new",
+            profile_available=True,
+            message="added",
+        )
+    )
+    db.commit()
+    run_id = run.id
     db.close()
 
     settings = Settings(admin_username="admin", admin_password="password", session_secret="session-secret")
@@ -301,42 +467,66 @@ def test_api_employees_and_stats_require_admin_session():
 
     employees = client.get("/api/employees", params={"q": "Alpha", "has_email": True})
     stats = client.get("/api/stats")
+    run_details = client.get(f"/api/crawl-runs/{run_id}")
 
     assert employees.status_code == 200
     assert employees.json()["total"] == 1
     assert stats.status_code == 200
     assert stats.json()["new_in_last_run"] == 1
+    assert run_details.status_code == 200
+    assert run_details.json()["changes"]["new"][0]["full_name"] == "Alpha Person"
 
     app.dependency_overrides.clear()
 
 
-def _oauth_settings() -> Settings:
+def test_admin_refresh_employee_route_updates_only_requested_employee(monkeypatch):
-    return Settings(
+    engine = create_engine(
-        mcp_auth_mode="oauth",
+        "sqlite:///:memory:",
-        mcp_resource_url="https://api.example.com/mcp",
+        connect_args={"check_same_thread": False},
-        mcp_oauth_issuer="https://auth.example.com",
+        poolclass=StaticPool,
-        mcp_oauth_audience="miem-mcp",
-        mcp_oauth_jwks_url="https://auth.example.com/.well-known/jwks.json",
-        session_secret="session-secret",
     )
+    Base.metadata.create_all(engine)
+    Session = sessionmaker(bind=engine)
+    db = Session()
+    db.add(
+        Employee(
+            profile_key="org_person:133709486",
+            profile_type="org_person",
+            profile_id="133709486",
+            canonical_url="https://www.hse.ru/org/persons/133709486",
+            full_name="Будков Юрий Алексеевич",
+            status="active",
+        )
+    )
+    db.commit()
+    employee_id = db.scalar(select(Employee.id))
+    db.close()
 
+    settings = Settings(admin_username="admin", admin_password="password", session_secret="session-secret")
 
-def _oauth_key_and_token(
+    def override_db():
-    *,
+        session = Session()
-    issuer: str = "https://auth.example.com",
+        try:
-    audience: str = "miem-mcp",
+            yield session
-    scope: str = "mcp:tools",
+        finally:
-    exp: int | None = None,
+            session.close()
-    public_key=None,
+
-):
+    calls = []
-    private_key = rsa.generate_private_key(public_exponent=65537, key_size=2048)
+
-    claims = {
+    def fake_refresh_employee(db, refreshed_employee, route_settings):
-        "iss": issuer,
+        calls.append((refreshed_employee.id, route_settings))
-        "aud": audience,
+        return SimpleNamespace(status="completed")
-        "scope": scope,
+
-        "sub": "mcp-client",
+    app.dependency_overrides[get_db] = override_db
-        "iat": int(time.time()),
+    app.dependency_overrides[get_settings] = lambda: settings
-        "exp": exp or int(time.time()) + 300,
+    monkeypatch.setattr("app.admin.refresh_employee", fake_refresh_employee)
-    }
+    client = TestClient(app)
-    token = jwt.encode(claims, private_key, algorithm="RS256", headers={"kid": "test-key"})
+    client.cookies.set(SESSION_COOKIE, sign_session("admin", settings))
-    return public_key or private_key.public_key(), token
+
+    response = client.post(f"/admin/employees/{employee_id}/refresh", follow_redirects=False)
+
+    assert response.status_code == 303
+    assert response.headers["location"] == f"/admin/employees/{employee_id}?refresh_status=success"
+    assert calls == [(employee_id, settings)]
+
+    app.dependency_overrides.clear()

tests/test_config.py

@@ -1,6 +1,3 @@
-import pytest
-from pydantic import ValidationError
-
 from app.config import Settings
 
 
@@ -14,8 +11,3 @@ def test_numeric_crawl_limit_is_parsed():
     settings = Settings(crawl_limit="25")
 
     assert settings.crawl_limit == 25
-
-
-def test_mcp_auth_mode_rejects_oauth_or_token_fallback():
-    with pytest.raises(ValidationError):
-        Settings(mcp_auth_mode="oauth_or_token")

tests/test_crawler.py

@@ -1,10 +1,55 @@
+import gzip
 from datetime import datetime, timezone
 
-from app.models import CrawlRun, Employee
+from app.models import CrawlError, CrawlRun, CrawlRunEmployeeChange, Employee, EmployeePublication, EmployeeSnapshot, ParseResourceCache
-from app.services.crawler import _mark_dismissed, _upsert_employee
+from app.services.crawler import _checksum, _mark_dismissed, _upsert_employee
+from app.services.resource_cache import ResourceCache
 
 
-def test_mark_dismissed_only_marks_missing_active_employees(db_session):
+class FakeResponse:
+    def __init__(self, status_code):
+        self.status_code = status_code
+
+
+class FakeSession:
+    def __init__(self, statuses):
+        self.statuses = statuses
+
+    def get(self, url, **_kwargs):
+        return FakeResponse(self.statuses[url])
+
+
+class ConditionalResponse:
+    def __init__(self, status_code, text="", headers=None):
+        self.status_code = status_code
+        self._text = text
+        self.headers = headers or {}
+        self.text_read = False
+
+    @property
+    def text(self):
+        self.text_read = True
+        return self._text
+
+    def raise_for_status(self):
+        return None
+
+
+class ConditionalSession:
+    def __init__(self):
+        self.requests = []
+        self.not_modified_response = ConditionalResponse(304)
+
+    def get(self, url, **kwargs):
+        self.requests.append((url, kwargs))
+        if kwargs["headers"].get("If-None-Match") == '"cached"':
+            return self.not_modified_response
+        return ConditionalResponse(200, "fresh", {"ETag": '"fresh"'})
+
+
+def test_mark_dismissed_records_missing_source_when_profile_is_available(db_session):
+    run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
+    db_session.add(run)
     db_session.add(
         Employee(
             profile_key="staff:kept",
@@ -16,8 +61,8 @@ def test_mark_dismissed_only_marks_missing_active_employees(db_session):
     )
     db_session.add(
         Employee(
-            profile_key="staff:gone",
+            profile_key="staff:missing",
-            canonical_url="https://www.hse.ru/staff/gone",
+            canonical_url="https://www.hse.ru/staff/missing",
             status="active",
             first_seen_at=datetime.now(timezone.utc),
             last_seen_at=datetime.now(timezone.utc),
@@ -25,16 +70,53 @@ def test_mark_dismissed_only_marks_missing_active_employees(db_session):
     )
     db_session.commit()
 
-    dismissed = _mark_dismissed(db_session, {"staff:kept"})
+    dismissed = _mark_dismissed(
+        db_session,
+        run,
+        {"staff:kept"},
+        FakeSession({"https://www.hse.ru/staff/missing": 200}),
+        30,
+    )
+
+    assert dismissed == 0
+    assert db_session.query(Employee).filter_by(profile_key="staff:kept").one().status == "active"
+    missing = db_session.query(Employee).filter_by(profile_key="staff:missing").one()
+    assert missing.status == "active"
+    assert missing.dismissed_at is None
+    change = db_session.query(CrawlRunEmployeeChange).one()
+    assert change.change_type == "missing_from_source"
+    assert change.profile_available is True
+
+
+def test_mark_dismissed_marks_missing_employee_when_profile_is_unavailable(db_session):
+    run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
+    employee = Employee(
+        profile_key="staff:gone",
+        canonical_url="https://www.hse.ru/staff/gone",
+        status="active",
+        first_seen_at=datetime.now(timezone.utc),
+        last_seen_at=datetime.now(timezone.utc),
+    )
+    db_session.add_all([run, employee])
+    db_session.commit()
+
+    dismissed = _mark_dismissed(
+        db_session,
+        run,
+        set(),
+        FakeSession({"https://www.hse.ru/staff/gone": 404}),
+        30,
+    )
 
     assert dismissed == 1
-    assert db_session.query(Employee).filter_by(profile_key="staff:kept").one().status == "active"
+    assert employee.status == "dismissed"
-    gone = db_session.query(Employee).filter_by(profile_key="staff:gone").one()
+    assert employee.dismissed_at is not None
-    assert gone.status == "dismissed"
+    change = db_session.query(CrawlRunEmployeeChange).one()
-    assert gone.dismissed_at is not None
+    assert change.change_type == "dismissed"
+    assert change.profile_available is False
 
 
-def test_upsert_employee_increments_new_count_for_new_employee(db_session):
+def test_upsert_employee_increments_new_count_and_records_change_for_new_employee(db_session):
     run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
     db_session.add(run)
     db_session.commit()
@@ -56,3 +138,179 @@ def test_upsert_employee_increments_new_count_for_new_employee(db_session):
     db_session.commit()
 
     assert run.new_count == 1
+    change = db_session.query(CrawlRunEmployeeChange).one()
+    assert change.change_type == "new"
+    assert change.full_name == "New Person"
+
+
+def test_resource_cache_uses_etag_and_reuses_cached_body_on_304(db_session):
+    db_session.add(
+        ParseResourceCache(
+            profile_key="staff:cached",
+            resource_key="main-html",
+            method="GET",
+            url="https://www.hse.ru/staff/cached",
+            request_fingerprint="020d59db7b358d9023d0f185bcbf5a9c085d3cf2bf91d92d48eee9147e8d0f01",
+            etag='"cached"',
+            body_hash="cached-hash",
+            body_snapshot=gzip.compress("cached body".encode("utf-8")),
+            parser_version="0.6.0",
+        )
+    )
+    db_session.commit()
+    session = ConditionalSession()
+
+    result = ResourceCache(db_session).fetch_text(
+        session,
+        profile_key="staff:cached",
+        resource_key="main-html",
+        method="GET",
+        url="https://www.hse.ru/staff/cached",
+        headers={"User-Agent": "test"},
+        timeout=10,
+    )
+
+    assert session.requests[0][1]["headers"]["If-None-Match"] == '"cached"'
+    assert result.text == "cached body"
+    assert result.from_cache is True
+    assert session.not_modified_response.text_read is False
+
+
+def test_upsert_employee_skips_snapshot_when_checksum_is_unchanged(db_session):
+    first_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
+    second_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
+    db_session.add_all([first_run, second_run])
+    db_session.commit()
+
+    _, first_changed = _upsert_employee(db_session, first_run, _parsed_employee("same"))
+    _, second_changed = _upsert_employee(db_session, second_run, _parsed_employee("same"))
+    db_session.commit()
+
+    assert first_changed is True
+    assert second_changed is False
+    assert db_session.query(EmployeeSnapshot).count() == 1
+
+
+def test_upsert_employee_saves_publications_and_reuses_existing_rows(db_session):
+    first_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
+    second_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
+    db_session.add_all([first_run, second_run])
+    db_session.commit()
+
+    parsed = _parsed_employee("published")
+    parsed["sections"] = [
+        {
+            "type": "publications",
+            "publications": [
+                {
+                    "id": "888959076",
+                    "publication_id": "888959076",
+                    "title": "Detailed Publication",
+                    "year": 2023,
+                    "publication_type": "ARTICLE",
+                    "language": "ru",
+                    "status": 1,
+                    "url": "https://publications.hse.ru/view/888959076",
+                    "doi_url": "https://doi.org/10.1/test",
+                    "citation_text": "Detailed citation",
+                    "annotation": {"ru": "Аннотация"},
+                    "description": {"main": "Detailed citation"},
+                    "authors": [{"id": "1", "title_ru": "Автор"}],
+                    "raw_data": {"id": "888959076", "title": "Detailed Publication"},
+                }
+            ],
+        }
+    ]
+
+    employee, _ = _upsert_employee(db_session, first_run, parsed)
+    db_session.commit()
+    _upsert_employee(db_session, second_run, _parsed_employee_with_publication("published"))
+    db_session.commit()
+
+    publications = db_session.query(EmployeePublication).filter_by(employee_id=employee.id).all()
+    assert len(publications) == 1
+    assert publications[0].doi_url == "https://doi.org/10.1/test"
+    assert publications[0].authors == [{"id": "1", "title_ru": "Автор"}]
+
+
+def test_upsert_employee_records_publication_errors_without_failing_employee(monkeypatch, db_session):
+    run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
+    db_session.add(run)
+    db_session.commit()
+
+    def broken_sync(*_args, **_kwargs):
+        raise RuntimeError("boom")
+
+    monkeypatch.setattr("app.services.crawler._sync_employee_publications", broken_sync)
+
+    employee, changed = _upsert_employee(db_session, run, _parsed_employee_with_publication("error-safe"))
+    db_session.commit()
+
+    assert changed is True
+    assert employee.full_name == "Same Person"
+    assert db_session.query(Employee).filter_by(profile_key="staff:error-safe").one()
+    error = db_session.query(CrawlError).one()
+    assert "публикации" in error.message.lower()
+
+
+def test_checksum_changes_when_widget_data_changes():
+    base = _parsed_employee("widgets")
+    changed = _parsed_employee("widgets")
+    changed["sections"] = [
+        {
+            "type": "publications",
+            "publications": [{"id": "1", "title": "New publication"}],
+        }
+    ]
+
+    assert _checksum(base) != _checksum(changed)
+
+
+def test_checksum_ignores_date_dependent_experience_text():
+    first = _parsed_employee("experience")
+    second = _parsed_employee("experience")
+    first["sections"] = [{"raw_text": "Стаж работы в НИУ ВШЭ: 5 лет"}]
+    second["sections"] = [{"raw_text": "Стаж работы в НИУ ВШЭ: 6 лет"}]
+
+    assert _checksum(first) == _checksum(second)
+
+
+def _parsed_employee(profile_id: str) -> dict:
+    return {
+        "source_url": f"https://www.hse.ru/staff/{profile_id}",
+        "profile_type": "staff",
+        "profile_id": profile_id,
+        "full_name": "Same Person",
+        "tabs": [],
+        "sections": [],
+        "parser_version": "0.6.0",
+        "_html": "<html></html>",
+    }
+
+
+def _parsed_employee_with_publication(profile_id: str) -> dict:
+    parsed = _parsed_employee(profile_id)
+    parsed["sections"] = [
+        {
+            "type": "publications",
+            "publications": [
+                {
+                    "id": "888959076",
+                    "publication_id": "888959076",
+                    "title": "Detailed Publication",
+                    "year": 2023,
+                    "publication_type": "ARTICLE",
+                    "language": "ru",
+                    "status": 1,
+                    "url": "https://publications.hse.ru/view/888959076",
+                    "doi_url": "https://doi.org/10.1/test",
+                    "citation_text": "Detailed citation",
+                    "annotation": {"ru": "Аннотация"},
+                    "description": {"main": "Detailed citation"},
+                    "authors": [{"id": "1", "title_ru": "Автор"}],
+                    "raw_data": {"id": "888959076", "title": "Detailed Publication"},
+                }
+            ],
+        }
+    ]
+    return parsed

tests/test_dataset_versions.py

@@ -0,0 +1,88 @@
+from datetime import datetime, timezone
+
+from app.models import Employee
+from app.services.dataset_versions import get_or_create_current_version, sync_employees_payload
+
+
+def _employee(profile_key: str, checksum: str, *, status: str = "active") -> Employee:
+    return Employee(
+        profile_key=profile_key,
+        profile_type=profile_key.split(":", 1)[0],
+        profile_id=profile_key.split(":", 1)[1],
+        canonical_url=f"https://www.hse.ru/{profile_key}",
+        full_name=profile_key,
+        status=status,
+        first_seen_at=datetime.now(timezone.utc),
+        last_seen_at=datetime.now(timezone.utc),
+        current_data={"profile_key": profile_key},
+        current_checksum=checksum,
+    )
+
+
+def test_dataset_version_hash_is_stable_for_same_employee_state(db_session):
+    db_session.add(_employee("staff:alpha", "a" * 64))
+    db_session.commit()
+
+    first = get_or_create_current_version(db_session)
+    db_session.commit()
+    second = get_or_create_current_version(db_session)
+
+    assert second.id == first.id
+    assert second.hash == first.hash
+    assert second.employee_count == 1
+
+
+def test_dataset_version_hash_changes_when_employee_checksum_changes(db_session):
+    employee = _employee("staff:alpha", "a" * 64)
+    db_session.add(employee)
+    db_session.commit()
+    first = get_or_create_current_version(db_session)
+    db_session.commit()
+
+    employee.current_checksum = "b" * 64
+    db_session.commit()
+    second = get_or_create_current_version(db_session)
+
+    assert second.hash != first.hash
+    assert second.previous_hash == first.hash
+
+
+def test_sync_employees_diff_spans_multiple_intermediate_versions(db_session):
+    alpha = _employee("staff:alpha", "a" * 64)
+    db_session.add(alpha)
+    db_session.commit()
+    first = get_or_create_current_version(db_session)
+    db_session.commit()
+
+    beta = _employee("staff:beta", "b" * 64)
+    db_session.add(beta)
+    db_session.commit()
+    get_or_create_current_version(db_session)
+    db_session.commit()
+
+    alpha.current_checksum = "c" * 64
+    alpha.current_data = {"profile_key": "staff:alpha", "changed": True}
+    db_session.commit()
+
+    payload = sync_employees_payload(db_session, client_hash=first.hash, include_data=False)
+
+    assert payload["mode"] == "delta"
+    assert [item["profile_key"] for item in payload["changes"]["added"]] == ["staff:beta"]
+    assert [item["profile_key"] for item in payload["changes"]["updated"]] == ["staff:alpha"]
+    assert payload["changes"]["dismissed"] == []
+    assert payload["changes"]["removed"] == []
+
+
+def test_sync_employees_reports_dismissed_as_tombstone(db_session):
+    alpha = _employee("staff:alpha", "a" * 64)
+    db_session.add(alpha)
+    db_session.commit()
+    first = get_or_create_current_version(db_session)
+    db_session.commit()
+
+    alpha.status = "dismissed"
+    db_session.commit()
+    payload = sync_employees_payload(db_session, client_hash=first.hash, include_data=False)
+
+    assert payload["changes"]["dismissed"][0]["profile_key"] == "staff:alpha"
+    assert payload["changes"]["dismissed"][0]["status"] == "dismissed"

tests/test_db_schema.py

@@ -0,0 +1,71 @@
+from sqlalchemy import create_engine, inspect, text
+
+from app.db import _ensure_runtime_schema
+
+
+def test_runtime_schema_adds_skipped_count_to_existing_crawl_runs_table(monkeypatch):
+    engine = create_engine("sqlite:///:memory:")
+    with engine.begin() as connection:
+        connection.execute(
+            text(
+                """
+                CREATE TABLE crawl_runs (
+                  id INTEGER PRIMARY KEY,
+                  source_url TEXT NOT NULL,
+                  status VARCHAR(32) NOT NULL DEFAULT 'running',
+                  found_count INTEGER NOT NULL DEFAULT 0,
+                  parsed_count INTEGER NOT NULL DEFAULT 0
+                )
+                """
+            )
+        )
+    monkeypatch.setattr("app.db.engine", engine)
+
+    _ensure_runtime_schema()
+
+    columns = {column["name"] for column in inspect(engine).get_columns("crawl_runs")}
+    assert "skipped_count" in columns
+
+
+def test_runtime_schema_creates_employee_publications_table_when_employees_exist(monkeypatch):
+    engine = create_engine("sqlite:///:memory:")
+    with engine.begin() as connection:
+        connection.execute(
+            text(
+                """
+                CREATE TABLE employees (
+                  id INTEGER PRIMARY KEY,
+                  profile_key VARCHAR(255) NOT NULL UNIQUE,
+                  canonical_url TEXT NOT NULL,
+                  status VARCHAR(32) NOT NULL DEFAULT 'active',
+                  first_seen_at DATETIME NOT NULL,
+                  last_seen_at DATETIME NOT NULL,
+                  created_at DATETIME NOT NULL,
+                  updated_at DATETIME NOT NULL
+                )
+                """
+            )
+        )
+        connection.execute(
+            text(
+                """
+                CREATE TABLE crawl_runs (
+                  id INTEGER PRIMARY KEY,
+                  source_url TEXT NOT NULL,
+                  status VARCHAR(32) NOT NULL DEFAULT 'running',
+                  found_count INTEGER NOT NULL DEFAULT 0,
+                  parsed_count INTEGER NOT NULL DEFAULT 0,
+                  skipped_count INTEGER NOT NULL DEFAULT 0
+                )
+                """
+            )
+        )
+    monkeypatch.setattr("app.db.engine", engine)
+
+    _ensure_runtime_schema()
+    _ensure_runtime_schema()
+
+    inspector = inspect(engine)
+    assert "employee_publications" in inspector.get_table_names()
+    columns = {column["name"] for column in inspector.get_columns("employee_publications")}
+    assert {"employee_id", "publication_id", "doi_url", "authors", "raw_data", "source_hash"}.issubset(columns)

tests/test_employee_detail_template.py

@@ -27,4 +27,6 @@ def test_employee_detail_template_is_human_readable():
     assert "Дата увольнения" in template
     assert "Тип профиля" in template
     assert "ID профиля" in template
+    assert "Обновить данные" in template
+    assert 'action="/admin/employees/{{ employee.id }}/refresh"' in template
     assert "Снапшоты" in template

tests/test_parser.py

@@ -1,6 +1,6 @@
 from bs4 import BeautifulSoup
 
-from app.parser.profile import enrich_sections_from_hse_widgets, extract_person_tabs
+from app.parser.profile import enrich_sections_from_hse_widgets, extract_person_tabs, extract_sections
 from app.parser.profile_url import normalize_profile_url, parse_profile_identity
 
 
@@ -34,7 +34,21 @@ class FakeSession:
                             "type": "ARTICLE",
                             "title": "Дублирование пакетов",
                             "year": 2023,
+                            "language": {"name": "ru"},
+                            "status": 1,
+                            "authorsByType": {
+                                "author": [
+                                    {
+                                        "id": "568398853",
+                                        "href": "/org/persons/568398853",
+                                        "title": {"ru": "Левицкий И. А.", "en": ""},
+                                        "reverseTitle": {"ru": "И. А. Левицкий", "en": ""},
+                                    }
+                                ]
+                            },
                             "description": {"short": {"ru": "Информационные процессы. 2023."}},
+                            "annotation": {"ru": "<p>Русская аннотация</p>"},
+                            "documents": {"DOI": {"href": "https://doi.org/10.1/test"}},
                         }
                     ],
                 },
@@ -64,6 +78,47 @@ class FakeSession:
         )
 
 
+class GroupedPublicationsSession(FakeSession):
+    def post(self, url, **kwargs):
+        self.posts.append((url, kwargs))
+        return FakeResponse(
+            {
+                "status": "ok",
+                "result": {
+                    "more": False,
+                    "total": 1,
+                    "groupType": 2,
+                    "items": {
+                        "year": {
+                            "header": {"ru": "по году", "en": "by year"},
+                            "criteria": {"year": []},
+                            "items": {
+                                "2011": [
+                                    {
+                                        "id": "146366790",
+                                        "type": "ARTICLE",
+                                        "title": "Развитие теории самосогласованного поля",
+                                        "year": 2011,
+                                        "description": {"short": {"ru": "Журнал физической химии 2011."}},
+                                    }
+                                ],
+                                "2012": [
+                                    {
+                                        "id": "146367323",
+                                        "type": "ARTICLE",
+                                        "title": "Self-consistent field theory investigation",
+                                        "year": 2012,
+                                        "description": {"short": {"en": "Russian Journal of Physical Chemistry A 2012."}},
+                                    }
+                                ],
+                            },
+                        }
+                    },
+                },
+            }
+        )
+
+
 def test_normalize_profile_url_supports_staff_and_org_persons():
     assert normalize_profile_url("/staff/avsergeev#sci") == "https://www.hse.ru/staff/avsergeev"
     assert normalize_profile_url("https://www.hse.ru/org/persons/123/") == "https://www.hse.ru/org/persons/123"
@@ -112,8 +167,68 @@ def test_enrich_sections_from_hse_widgets_loads_publications_and_vkr():
 
     assert publications["publications_count"] == 1
     assert publications["publications"][0]["url"] == "https://publications.hse.ru/view/888959076"
+    assert publications["publications"][0]["doi_url"] == "https://doi.org/10.1/test"
+    assert publications["publications"][0]["annotation"] == {"ru": "Русская аннотация"}
+    assert publications["publications"][0]["authors"][0]["is_current_employee"] is True
     assert theses["theses_count"] == 1
     assert theses["theses"][0]["student"] == "Лесняк Владислав Евгеньевич"
     assert theses["theses"][0]["project_url"] == "https://www.hse.ru/edu/vkr/1045750164"
     assert session.posts[0][0] == "https://publications.hse.ru/api/searchPubs"
     assert session.gets[0][1]["params"] == {"supervisorId": "803294906"}
+
+
+def test_enrich_sections_from_hse_widgets_loads_grouped_publications():
+    soup = BeautifulSoup(
+        """
+        <script src="/n/stat/publications/dist-w/publs.js" data-author="133709486" data-widget-name="AuthorSearch"></script>
+        """,
+        "html.parser",
+    )
+    session = GroupedPublicationsSession()
+
+    sections = enrich_sections_from_hse_widgets(
+        session,
+        soup,
+        "https://www.hse.ru/org/persons/133709486",
+        {"User-Agent": "test"},
+        10,
+        [],
+    )
+
+    publications = next(section for section in sections if section["type"] == "publications")
+
+    assert publications["publications_count"] == 2
+    assert [item["id"] for item in publications["publications"]] == ["146366790", "146367323"]
+    assert publications["publications"][0]["url"] == "https://publications.hse.ru/view/146366790"
+    assert publications["publications"][1]["url"] == "https://publications.hse.ru/view/146367323"
+
+
+def test_news_heading_with_publications_word_does_not_absorb_widget_publications():
+    soup = BeautifulSoup(
+        """
+        <h2>Статья профессора МИЭМ вошла в число самых популярных публикаций на портале SpringerLink</h2>
+        <div class="post__text">
+          <p>Первоначально статья профессора вышла в российском журнале.</p>
+        </div>
+        <script src="/n/stat/publications/dist-w/publs.js" data-author="133709486" data-widget-name="AuthorSearch"></script>
+        """,
+        "html.parser",
+    )
+    session = FakeSession()
+
+    sections = extract_sections(soup, "https://www.hse.ru/org/persons/133709486")
+    sections = enrich_sections_from_hse_widgets(
+        session,
+        soup,
+        "https://www.hse.ru/org/persons/133709486",
+        {"User-Agent": "test"},
+        10,
+        sections,
+    )
+
+    assert sections[0]["type"] == "paragraphs"
+    assert sections[0]["title"].startswith("Статья профессора")
+    publications = [section for section in sections if section["type"] == "publications"]
+    assert len(publications) == 1
+    assert publications[0]["title"] == "Публикации и исследования"
+    assert publications[0]["publications_count"] == 1