diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..c36db1a --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,5 @@ +# Changelog + +## 0.7.1 + +- Добавлена кнопка «Проверить уволенных» для принудительной сверки статуса уволенных сотрудников с текущим списком источника. diff --git a/README.md b/README.md index 094e265..2fb00f0 100644 --- a/README.md +++ b/README.md @@ -1,142 +1,150 @@ -# MIEM Employees Server - -Сервис собирает сотрудников МИЭМ с сайта ВШЭ, хранит карточки и историю обновлений в Postgres, показывает минимальную админку и отдает read-only MCP endpoint для ИИ-агентов. - -## Архитектура - -- `api`: FastAPI, REST API, HTML-админка, healthcheck. -- `worker`: weekly scheduler, который запускает парсинг по `CRAWL_CRON`. -- `mcp`: открытый HTTP MCP endpoint для ИИ-агентов. -- `postgres`: основная БД. - -Парсер использует фиксированный источник сотрудников, по умолчанию `https://miem.hse.ru/persons`. Для каждой карточки сохраняются ФИО, должности, год начала работы, контакты, идентификаторы, вкладки профиля, секции, публикации, курсы, ВКР, новости, JSON-снапшот и сжатый HTML-снапшот. Детальные публикации дополнительно нормализуются в отдельную таблицу `employee_publications`, а новости из блока «В новостях» — в `employee_news_links`. Ссылки обходятся только из меню профиля самого сотрудника (`person-menu`), например `#sci`, `#teaching`, `#main`. - -## Переменные окружения - -Скопируйте `.env.example` в `.env` и поменяйте секреты: - -```bash -cp .env.example .env -``` - -Основные настройки: - -- `DATABASE_URL`: строка подключения SQLAlchemy. -- `SOURCE_URL`: список сотрудников МИЭМ. -- `CRAWL_CRON`: расписание в формате crontab, по умолчанию `0 3 * * 1`. -- `CRAWL_LIMIT`: опциональный лимит профилей для тестового запуска. -- `ADMIN_USERNAME`, `ADMIN_PASSWORD`: логин и пароль админки. -- `SESSION_SECRET`: секрет подписи cookie. -- `PARSER_USE_PLAYWRIGHT`: включение Playwright-рендера динамических вкладок. - -## Локальный запуск - -```bash -python -m venv .venv -.venv\Scripts\activate -pip install -r requirements.txt -uvicorn app.main:app --reload -``` - -Админка: `http://localhost:8000/admin`. - -В админке доступны: - -- `Dashboard`: общая статистика, последний добавленный сотрудник, прогресс текущего/последнего парсинга и ручной запуск. -- `Directory`: настраиваемая таблица сотрудников с фильтрами, сортировкой, пагинацией и выбором колонок. -- `Runs`: история запусков, ошибки и progress bar. - -## Docker Compose - -```bash -docker compose up --build -``` - -По умолчанию: - -- API и админка: `http://localhost:8000` -- MCP: `http://localhost:8001/mcp` -- Postgres: `localhost:5432` - -Таблицы создаются приложением при старте. При обновлении существующей базы приложение также добавляет недостающие runtime-колонки, например `crawl_runs.skipped_count`. SQL-миграции для ручного применения лежат в `migrations/`. - -## Наполнение БД - -Основная карточка сотрудника хранится в `employees`: профиль, статус, даты обнаружения/увольнения, текущий JSON `current_data`, checksum и версия парсера. История успешных изменений сохраняется в `employee_snapshots` вместе с JSON-снимком и сжатым HTML профиля. - -Публикации теперь хранятся в двух видах: - -- краткий список остается внутри `employees.current_data.sections[].publications` для обратной совместимости; -- детальные записи сохраняются в `employee_publications` и связываются с сотрудником через `employee_id`. - -`employee_publications` содержит `publication_id`, название, год, тип публикации, язык, статус, ссылку на карточку HSE Publications, DOI, внешние/document-ссылки, citation text, аннотацию, описание, авторов, raw JSON ответа `searchPubs` и `source_hash` для безопасного повторного upsert. Уникальность поддерживается по `(employee_id, publication_id)` и `(employee_id, source_hash)`, поэтому повторный crawl не должен создавать дубликаты. - -`list_employee_publications` сначала читает `employee_publications`; если детальных строк еще нет, возвращает старые публикации из `current_data`. - -Новости сотрудников также хранятся в двух видах: - -- краткий список остается внутри `employees.current_data.sections[].news_links`; -- нормализованные карточки из вкладки «В новостях» сохраняются в `employee_news_links`. - -`employee_news_links` содержит название новости, ссылку, краткое описание, дату публикации, год публикации, raw JSON карточки и `source_hash`. Уникальность поддерживается по `(employee_id, url)` и `(employee_id, source_hash)`, поэтому повторный crawl не создает дубликаты. - -## Парсинг - -Weekly worker запускается по `CRAWL_CRON`. Ручной запуск доступен в админке на `Dashboard` и странице `Runs` или через REST: - -```bash -curl -X POST http://localhost:8000/api/crawl-runs --cookie "miem_admin_session=..." -``` - -Алгоритм обновления: - -- найденные сотрудники получают статус `active` и обновленный `last_seen_at`; -- новые сотрудники добавляются в `employees`; -- количество новых сотрудников за запуск сохраняется в `crawl_runs.new_count`; -- публикации из HSE Publications записываются в `employee_publications`, а краткий список остается в JSON профиля; -- новости из блока «В новостях» записываются в `employee_news_links`, а краткий список остается в JSON профиля; -- активные сотрудники, исчезнувшие из текущего списка источника, получают статус `dismissed` и `dismissed_at`; +# MIEM Employees Server + +Сервис собирает сотрудников МИЭМ с сайта ВШЭ, хранит карточки и историю обновлений в Postgres, показывает минимальную админку и отдает read-only MCP endpoint для ИИ-агентов. + +## Архитектура + +- `api`: FastAPI, REST API, HTML-админка, healthcheck. +- `worker`: weekly scheduler, который запускает парсинг по `CRAWL_CRON`. +- `mcp`: открытый HTTP MCP endpoint для ИИ-агентов. +- `postgres`: основная БД. + +Парсер использует фиксированный источник сотрудников, по умолчанию `https://miem.hse.ru/persons`. Для каждой карточки сохраняются ФИО, должности, год начала работы, контакты, идентификаторы, вкладки профиля, секции, публикации, курсы, ВКР, новости, JSON-снапшот и сжатый HTML-снапшот. Детальные публикации дополнительно нормализуются в отдельную таблицу `employee_publications`, а новости из блока «В новостях» — в `employee_news_links`. Ссылки обходятся только из меню профиля самого сотрудника (`person-menu`), например `#sci`, `#teaching`, `#main`. + +## Переменные окружения + +Скопируйте `.env.example` в `.env` и поменяйте секреты: + +```bash +cp .env.example .env +``` + +Основные настройки: + +- `DATABASE_URL`: строка подключения SQLAlchemy. +- `SOURCE_URL`: список сотрудников МИЭМ. +- `CRAWL_CRON`: расписание в формате crontab, по умолчанию `0 3 * * 1`. +- `CRAWL_LIMIT`: опциональный лимит профилей для тестового запуска. +- `ADMIN_USERNAME`, `ADMIN_PASSWORD`: логин и пароль админки. +- `SESSION_SECRET`: секрет подписи cookie. +- `PARSER_USE_PLAYWRIGHT`: включение Playwright-рендера динамических вкладок. +- `DISMISSAL_CONFIRMATION_RUNS`: сколько последовательных проверок недоступности нужно для увольнения, по умолчанию `3`. +- `MAX_AUTO_DISMISSALS_PER_RUN`: защитный лимит массовых автоматических увольнений за один запуск, по умолчанию `25`. + +## Локальный запуск + +```bash +python -m venv .venv +.venv\Scripts\activate +pip install -r requirements.txt +uvicorn app.main:app --reload +``` + +Админка: `http://localhost:8000/admin`. + +В админке доступны: + +- `Dashboard`: общая статистика, последний добавленный сотрудник, прогресс текущего/последнего парсинга и ручной запуск. +- `Directory`: настраиваемая таблица сотрудников с фильтрами, сортировкой, пагинацией и выбором колонок. +- `Runs`: история запусков, ошибки и progress bar. + +## Docker Compose + +```bash +docker compose up --build +``` + +По умолчанию: + +- API и админка: `http://localhost:8000` +- MCP: `http://localhost:8001/mcp` +- Postgres: `localhost:5432` + +Таблицы создаются приложением при старте. При обновлении существующей базы приложение также добавляет недостающие runtime-колонки, например `crawl_runs.skipped_count`. SQL-миграции для ручного применения лежат в `migrations/`. + +## Наполнение БД + +Основная карточка сотрудника хранится в `employees`: профиль, статус, даты обнаружения/увольнения, текущий JSON `current_data`, checksum и версия парсера. История успешных изменений сохраняется в `employee_snapshots` вместе с JSON-снимком и сжатым HTML профиля. + +Публикации теперь хранятся в двух видах: + +- краткий список остается внутри `employees.current_data.sections[].publications` для обратной совместимости; +- детальные записи сохраняются в `employee_publications` и связываются с сотрудником через `employee_id`. + +`employee_publications` содержит `publication_id`, название, год, тип публикации, язык, статус, ссылку на карточку HSE Publications, DOI, внешние/document-ссылки, citation text, аннотацию, описание, авторов, raw JSON ответа `searchPubs` и `source_hash` для безопасного повторного upsert. Уникальность поддерживается по `(employee_id, publication_id)` и `(employee_id, source_hash)`, поэтому повторный crawl не должен создавать дубликаты. + +`list_employee_publications` сначала читает `employee_publications`; если детальных строк еще нет, возвращает старые публикации из `current_data`. + +Новости сотрудников также хранятся в двух видах: + +- краткий список остается внутри `employees.current_data.sections[].news_links`; +- нормализованные карточки из вкладки «В новостях» сохраняются в `employee_news_links`. + +`employee_news_links` содержит название новости, ссылку, краткое описание, дату публикации, год публикации, raw JSON карточки и `source_hash`. Уникальность поддерживается по `(employee_id, url)` и `(employee_id, source_hash)`, поэтому повторный crawl не создает дубликаты. + +## Парсинг + +Weekly worker запускается по `CRAWL_CRON`. Ручной запуск доступен в админке на `Dashboard` и странице `Runs` или через REST: + +```bash +curl -X POST http://localhost:8000/api/crawl-runs --cookie "miem_admin_session=..." +``` + +Алгоритм обновления: + +- найденные сотрудники получают статус `active` и обновленный `last_seen_at`; +- новые сотрудники добавляются в `employees`; +- если профиль перенесен на другой URL, он сопоставляется с прежней записью по единственному точному совпадению ФИО; +- старые URL сохраняются в истории `employee_profile_urls`; +- количество новых сотрудников за запуск сохраняется в `crawl_runs.new_count`; +- публикации из HSE Publications записываются в `employee_publications`, а краткий список остается в JSON профиля; +- новости из блока «В новостях» записываются в `employee_news_links`, а краткий список остается в JSON профиля; +- один `404` старого профиля переводит сотрудника в статус `verification_required`, а не в `dismissed`; +- статус `dismissed` устанавливается только после нескольких последовательных проверок `404`/`410`; +- сетевые ошибки и ответы `5xx` не считаются подтверждением увольнения; +- если число кандидатов на увольнение превышает защитный лимит, автоматическое увольнение приостанавливается; +- кнопка «Проверить уволенных» сверяет только их profile_key с текущим списком источника и возвращает найденных сотрудников в `active` без обновления содержимого профиля; - каждый успешный новый или измененный разбор сохраняет запись в `employee_snapshots`; -- неизмененные профили учитываются в `crawl_runs.skipped_count` и не получают новый snapshot. - -Во время выполнения парсинга `found_count`, `parsed_count`, `skipped_count` и `error_count` обновляются в базе. Админка опрашивает `/api/crawl-runs/latest` и показывает прогресс как `(parsed_count + skipped_count + error_count) / found_count`. - -## MCP - -Endpoint: `POST /mcp`, без авторизации на уровне приложения. - -Поддерживаемые tools: - -- `get_service_info()` -- `sync_employees(client_hash?, include_data?)` -- `search_employees(query, status?, limit?)` -- `get_employee(profile_id_or_url)` -- `list_employee_publications(profile_id_or_url)` — публикации сотрудника; при наличии данных из `employee_publications` возвращает авторов, DOI, аннотацию, описание, citation text, год, тип, язык, статус и ссылку HSE Publications. -- `list_employee_courses(profile_id_or_url)` -- `get_crawl_status()` -- `get_crawl_run_details(run_id)` - -`get_service_info` возвращает метаданные сервиса, список tools и текущую версию набора сотрудников. `sync_employees` отдает полный snapshot или delta по `client_hash`; checksum набора строится по сотрудникам, их статусам и текущим checksums. Ответы tools возвращаются как JSON-строка внутри MCP `content[0].text`. - -Новости сотрудника отдельной MCP tool не имеют: они доступны в `get_employee(...).data.sections` и `sync_employees(include_data=true)` как секция `type = "news"` с массивом `news_links`. - -Пример локального запроса списка tools: - -```bash -curl http://localhost:8001/mcp \ - -H "Content-Type: application/json" \ - -d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}' -``` - -Если MCP нужно ограничить, делайте это на сетевом уровне: localhost binding, VPN, firewall, reverse proxy или другой внешний контур доступа. - -## Обслуживание - -```bash -docker compose logs -f api -docker compose logs -f worker -docker compose exec postgres pg_dump -U miem miem_workers > backup.sql -docker compose down -``` - -Версия сервиса: `0.7.0`. Админка всегда показывает версии backend и frontend в footer. +- неизмененные профили учитываются в `crawl_runs.skipped_count` и не получают новый snapshot. + +Во время выполнения парсинга `found_count`, `parsed_count`, `skipped_count` и `error_count` обновляются в базе. Админка опрашивает `/api/crawl-runs/latest` и показывает прогресс как `(parsed_count + skipped_count + error_count) / found_count`. + +## MCP + +Endpoint: `POST /mcp`, без авторизации на уровне приложения. + +Поддерживаемые tools: + +- `get_service_info()` +- `sync_employees(client_hash?, include_data?)` +- `search_employees(query, status?, limit?)` +- `get_employee(profile_id_or_url)` +- `list_employee_publications(profile_id_or_url)` — публикации сотрудника; при наличии данных из `employee_publications` возвращает авторов, DOI, аннотацию, описание, citation text, год, тип, язык, статус и ссылку HSE Publications. +- `list_employee_courses(profile_id_or_url)` +- `get_crawl_status()` +- `get_crawl_run_details(run_id)` + +`get_service_info` возвращает метаданные сервиса, список tools и текущую версию набора сотрудников. `sync_employees` отдает полный snapshot или delta по `client_hash`; checksum набора строится по сотрудникам, их статусам и текущим checksums. Ответы tools возвращаются как JSON-строка внутри MCP `content[0].text`. + +Новости сотрудника отдельной MCP tool не имеют: они доступны в `get_employee(...).data.sections` и `sync_employees(include_data=true)` как секция `type = "news"` с массивом `news_links`. + +Пример локального запроса списка tools: + +```bash +curl http://localhost:8001/mcp \ + -H "Content-Type: application/json" \ + -d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}' +``` + +Если MCP нужно ограничить, делайте это на сетевом уровне: localhost binding, VPN, firewall, reverse proxy или другой внешний контур доступа. + +## Обслуживание + +```bash +docker compose logs -f api +docker compose logs -f worker +docker compose exec postgres pg_dump -U miem miem_workers > backup.sql +docker compose down +``` + +Версия сервиса: `0.7.1`. Админка всегда показывает версии backend и frontend в footer. diff --git a/app/admin.py b/app/admin.py index 0d7c734..b313708 100644 --- a/app/admin.py +++ b/app/admin.py @@ -1,219 +1,238 @@ -from fastapi import APIRouter, BackgroundTasks, Depends, Form, Request -from fastapi.responses import HTMLResponse, RedirectResponse -from fastapi.templating import Jinja2Templates -from sqlalchemy import desc, func, select -from sqlalchemy.orm import Session - -from app.config import Settings, get_settings -from app.db import SessionLocal, get_db -from app.models import CrawlError, CrawlRun, Employee -from app.security import SESSION_COOKIE, require_admin, sign_session, verify_admin -from app.services.admin_data import ( - employee_detail_payload, - format_admin_datetime, - list_employees_page, - run_detail_payload, - run_payload, - stats_payload, -) -from app.services.crawl_control import get_running_run, run_crawl_if_idle -from app.services.crawler import refresh_employee -from app.version import BACKEND_VERSION, FRONTEND_VERSION - -router = APIRouter(prefix="/admin") -templates = Jinja2Templates(directory="app/templates") - - -@router.get("", response_class=HTMLResponse) -def dashboard(request: Request, db: Session = Depends(get_db), settings: Settings = Depends(get_settings)): - require_admin(request, settings) - counts = stats_payload(db) - counts["runs"] = db.scalar(select(func.count()).select_from(CrawlRun)) or 0 - counts["errors"] = db.scalar(select(func.count()).select_from(CrawlError)) or 0 - run_models = db.scalars(select(CrawlRun).order_by(desc(CrawlRun.started_at)).limit(5)).all() - runs = [run_payload(run) for run in run_models] - return _render(request, "dashboard.html", {"counts": counts, "runs": runs, "latest_run": runs[0] if runs else None}) - - -@router.get("/login", response_class=HTMLResponse) -def login_form(request: Request): - return _render(request, "login.html", {"error": None}) - - -@router.post("/login") -def login( - request: Request, - username: str = Form(...), - password: str = Form(...), - settings: Settings = Depends(get_settings), -): - if not verify_admin(username, password, settings): - return _render(request, "login.html", {"error": "Неверный логин или пароль"}, status_code=401) - redirect = RedirectResponse("/admin", status_code=303) - redirect.set_cookie(SESSION_COOKIE, sign_session(username, settings), httponly=True, samesite="lax") - return redirect - - -@router.post("/logout") -def logout(): - redirect = RedirectResponse("/admin/login", status_code=303) - redirect.delete_cookie(SESSION_COOKIE) - return redirect - - -@router.get("/employees", response_class=HTMLResponse) -def employees( - request: Request, - status: str | None = None, - q: str | None = None, - settings: Settings = Depends(get_settings), -): - require_admin(request, settings) - return RedirectResponse("/admin/directory", status_code=303) - - -@router.get("/directory", response_class=HTMLResponse) -def directory( - request: Request, - status: str | None = None, - q: str | None = None, - started_from: str | None = None, - started_to: str | None = None, - has_email: str | None = None, - sort: str = "full_name", - direction: str = "asc", - limit: int = 50, - offset: int = 0, - db: Session = Depends(get_db), - settings: Settings = Depends(get_settings), -): - require_admin(request, settings) - parsed_started_from = _parse_date(started_from) - parsed_started_to = _parse_date(started_to) - parsed_has_email = None if has_email in (None, "") else has_email == "true" - page = list_employees_page( - db, - status=status, - q=q, - started_from=parsed_started_from, - started_to=parsed_started_to, - has_email=parsed_has_email, - sort=sort, - direction=direction, - limit=limit, - offset=offset, - ) - return _render( - request, - "directory.html", - { - "page": page, - "filters": { - "status": status or "", - "q": q or "", - "started_from": started_from or "", - "started_to": started_to or "", - "has_email": has_email or "", - "sort": sort, - "direction": direction, - "limit": page["limit"], - "offset": offset, - }, - }, - ) - - -@router.get("/employees/{employee_id}", response_class=HTMLResponse) -def employee_detail( - employee_id: int, - request: Request, - db: Session = Depends(get_db), - settings: Settings = Depends(get_settings), -): - require_admin(request, settings) - employee = db.get(Employee, employee_id) - if not employee: - return RedirectResponse("/admin/employees", status_code=303) - snapshots = [ - { - "captured_display": format_admin_datetime(snapshot.captured_at), - "checksum": snapshot.checksum, - "parser_version": snapshot.parser_version, - } - for snapshot in sorted(employee.snapshots, key=lambda item: item.captured_at, reverse=True)[:20] - ] - return _render( - request, - "employee_detail.html", - { - "employee": employee, - "employee_view": employee_detail_payload(employee), - "snapshots": snapshots, - "refresh_status": request.query_params.get("refresh_status"), - }, - ) - - -@router.post("/employees/{employee_id}/refresh") -def refresh_employee_detail( - employee_id: int, - request: Request, - db: Session = Depends(get_db), - settings: Settings = Depends(get_settings), -): - require_admin(request, settings) - employee = db.get(Employee, employee_id) - if not employee: - return RedirectResponse("/admin/directory", status_code=303) - run = refresh_employee(db, employee, settings) - status = "success" if run.status == "completed" else "error" - return RedirectResponse(f"/admin/employees/{employee_id}?refresh_status={status}", status_code=303) - - -@router.get("/runs", response_class=HTMLResponse) -def runs(request: Request, db: Session = Depends(get_db), settings: Settings = Depends(get_settings)): - require_admin(request, settings) - run_models = db.scalars(select(CrawlRun).order_by(desc(CrawlRun.started_at)).limit(50)).all() - items = [run_payload(run) for run in run_models] - errors = db.scalars(select(CrawlError).order_by(desc(CrawlError.created_at)).limit(50)).all() - return _render(request, "runs.html", {"runs": items, "errors": errors}) - - -@router.get("/runs/{run_id}", response_class=HTMLResponse) -def run_detail( - run_id: int, - request: Request, - db: Session = Depends(get_db), - settings: Settings = Depends(get_settings), -): - require_admin(request, settings) - run = db.get(CrawlRun, run_id) - if not run: - return RedirectResponse("/admin/runs", status_code=303) - return _render(request, "run_detail.html", {"run": run_detail_payload(db, run)}) - - -@router.post("/runs") -def trigger_run( - request: Request, - background_tasks: BackgroundTasks, - db: Session = Depends(get_db), - settings: Settings = Depends(get_settings), -): - require_admin(request, settings) - if get_running_run(db): - return RedirectResponse("/admin/runs", status_code=303) - - def _crawl() -> None: - with SessionLocal() as db: - run_crawl_if_idle(db, settings) - - background_tasks.add_task(_crawl) - return RedirectResponse("/admin/runs", status_code=303) - - +from fastapi import APIRouter, BackgroundTasks, Depends, Form, Request +from fastapi.responses import HTMLResponse, RedirectResponse +from fastapi.templating import Jinja2Templates +from sqlalchemy import desc, func, select +from sqlalchemy.orm import Session + +from app.config import Settings, get_settings +from app.db import SessionLocal, get_db +from app.models import CrawlError, CrawlRun, Employee +from app.security import SESSION_COOKIE, require_admin, sign_session, verify_admin +from app.services.admin_data import ( + employee_detail_payload, + format_admin_datetime, + list_employees_page, + run_detail_payload, + run_payload, + stats_payload, +) +from app.services.crawl_control import get_running_run, run_crawl_if_idle +from app.services.crawler import refresh_dismissed_status, refresh_employee +from app.version import BACKEND_VERSION, FRONTEND_VERSION + +router = APIRouter(prefix="/admin") +templates = Jinja2Templates(directory="app/templates") + + +@router.get("", response_class=HTMLResponse) +def dashboard(request: Request, db: Session = Depends(get_db), settings: Settings = Depends(get_settings)): + require_admin(request, settings) + counts = stats_payload(db) + counts["runs"] = db.scalar(select(func.count()).select_from(CrawlRun)) or 0 + counts["errors"] = db.scalar(select(func.count()).select_from(CrawlError)) or 0 + run_models = db.scalars(select(CrawlRun).order_by(desc(CrawlRun.started_at)).limit(5)).all() + runs = [run_payload(run) for run in run_models] + return _render(request, "dashboard.html", {"counts": counts, "runs": runs, "latest_run": runs[0] if runs else None}) + + +@router.get("/login", response_class=HTMLResponse) +def login_form(request: Request): + return _render(request, "login.html", {"error": None}) + + +@router.post("/login") +def login( + request: Request, + username: str = Form(...), + password: str = Form(...), + settings: Settings = Depends(get_settings), +): + if not verify_admin(username, password, settings): + return _render(request, "login.html", {"error": "Неверный логин или пароль"}, status_code=401) + redirect = RedirectResponse("/admin", status_code=303) + redirect.set_cookie(SESSION_COOKIE, sign_session(username, settings), httponly=True, samesite="lax") + return redirect + + +@router.post("/logout") +def logout(): + redirect = RedirectResponse("/admin/login", status_code=303) + redirect.delete_cookie(SESSION_COOKIE) + return redirect + + +@router.get("/employees", response_class=HTMLResponse) +def employees( + request: Request, + status: str | None = None, + q: str | None = None, + settings: Settings = Depends(get_settings), +): + require_admin(request, settings) + return RedirectResponse("/admin/directory", status_code=303) + + +@router.get("/directory", response_class=HTMLResponse) +def directory( + request: Request, + status: str | None = None, + q: str | None = None, + started_from: str | None = None, + started_to: str | None = None, + has_email: str | None = None, + sort: str = "full_name", + direction: str = "asc", + limit: int = 50, + offset: int = 0, + db: Session = Depends(get_db), + settings: Settings = Depends(get_settings), +): + require_admin(request, settings) + parsed_started_from = _parse_date(started_from) + parsed_started_to = _parse_date(started_to) + parsed_has_email = None if has_email in (None, "") else has_email == "true" + page = list_employees_page( + db, + status=status, + q=q, + started_from=parsed_started_from, + started_to=parsed_started_to, + has_email=parsed_has_email, + sort=sort, + direction=direction, + limit=limit, + offset=offset, + ) + return _render( + request, + "directory.html", + { + "page": page, + "filters": { + "status": status or "", + "q": q or "", + "started_from": started_from or "", + "started_to": started_to or "", + "has_email": has_email or "", + "sort": sort, + "direction": direction, + "limit": page["limit"], + "offset": offset, + }, + }, + ) + + +@router.get("/employees/{employee_id}", response_class=HTMLResponse) +def employee_detail( + employee_id: int, + request: Request, + db: Session = Depends(get_db), + settings: Settings = Depends(get_settings), +): + require_admin(request, settings) + employee = db.get(Employee, employee_id) + if not employee: + return RedirectResponse("/admin/employees", status_code=303) + snapshots = [ + { + "captured_display": format_admin_datetime(snapshot.captured_at), + "checksum": snapshot.checksum, + "parser_version": snapshot.parser_version, + } + for snapshot in sorted(employee.snapshots, key=lambda item: item.captured_at, reverse=True)[:20] + ] + return _render( + request, + "employee_detail.html", + { + "employee": employee, + "employee_view": employee_detail_payload(employee), + "snapshots": snapshots, + "refresh_status": request.query_params.get("refresh_status"), + }, + ) + + +@router.post("/employees/{employee_id}/refresh") +def refresh_employee_detail( + employee_id: int, + request: Request, + db: Session = Depends(get_db), + settings: Settings = Depends(get_settings), +): + require_admin(request, settings) + employee = db.get(Employee, employee_id) + if not employee: + return RedirectResponse("/admin/directory", status_code=303) + run = refresh_employee(db, employee, settings) + status = "success" if run.status == "completed" else "error" + return RedirectResponse(f"/admin/employees/{employee_id}?refresh_status={status}", status_code=303) + + +@router.get("/runs", response_class=HTMLResponse) +def runs(request: Request, db: Session = Depends(get_db), settings: Settings = Depends(get_settings)): + require_admin(request, settings) + run_models = db.scalars(select(CrawlRun).order_by(desc(CrawlRun.started_at)).limit(50)).all() + items = [run_payload(run) for run in run_models] + errors = db.scalars(select(CrawlError).order_by(desc(CrawlError.created_at)).limit(50)).all() + return _render(request, "runs.html", {"runs": items, "errors": errors}) + + +@router.get("/runs/{run_id}", response_class=HTMLResponse) +def run_detail( + run_id: int, + request: Request, + db: Session = Depends(get_db), + settings: Settings = Depends(get_settings), +): + require_admin(request, settings) + run = db.get(CrawlRun, run_id) + if not run: + return RedirectResponse("/admin/runs", status_code=303) + return _render(request, "run_detail.html", {"run": run_detail_payload(db, run)}) + + +@router.post("/runs") +def trigger_run( + request: Request, + background_tasks: BackgroundTasks, + db: Session = Depends(get_db), + settings: Settings = Depends(get_settings), +): + require_admin(request, settings) + if get_running_run(db): + return RedirectResponse("/admin/runs", status_code=303) + + def _crawl() -> None: + with SessionLocal() as db: + run_crawl_if_idle(db, settings) + + background_tasks.add_task(_crawl) + return RedirectResponse("/admin/runs", status_code=303) + + @router.post("/crawl-now") def crawl_now( + request: Request, + background_tasks: BackgroundTasks, + db: Session = Depends(get_db), + settings: Settings = Depends(get_settings), +): + require_admin(request, settings) + if get_running_run(db): + return RedirectResponse("/admin", status_code=303) + + def _crawl() -> None: + with SessionLocal() as db: + run_crawl_if_idle(db, settings) + + background_tasks.add_task(_crawl) + return RedirectResponse("/admin", status_code=303) + + +@router.post("/dismissed/refresh") +def refresh_dismissed( request: Request, background_tasks: BackgroundTasks, db: Session = Depends(get_db), @@ -223,30 +242,30 @@ def crawl_now( if get_running_run(db): return RedirectResponse("/admin", status_code=303) - def _crawl() -> None: + def _refresh() -> None: with SessionLocal() as db: - run_crawl_if_idle(db, settings) + refresh_dismissed_status(db, settings) - background_tasks.add_task(_crawl) + background_tasks.add_task(_refresh) return RedirectResponse("/admin", status_code=303) def _render(request: Request, template: str, context: dict, status_code: int = 200) -> HTMLResponse: - payload = { - "request": request, - "backend_version": BACKEND_VERSION, - "frontend_version": FRONTEND_VERSION, - **context, - } - return templates.TemplateResponse(request, template, payload, status_code=status_code) - - -def _parse_date(value: str | None): - if not value: - return None - try: - from datetime import date - - return date.fromisoformat(value) - except ValueError: - return None + payload = { + "request": request, + "backend_version": BACKEND_VERSION, + "frontend_version": FRONTEND_VERSION, + **context, + } + return templates.TemplateResponse(request, template, payload, status_code=status_code) + + +def _parse_date(value: str | None): + if not value: + return None + try: + from datetime import date + + return date.fromisoformat(value) + except ValueError: + return None diff --git a/app/services/crawler.py b/app/services/crawler.py index 1a11ea7..46612df 100644 --- a/app/services/crawler.py +++ b/app/services/crawler.py @@ -1,133 +1,149 @@ -import gzip -import hashlib -import json -import re -import time -from datetime import datetime, timezone - -import requests -from sqlalchemy import inspect, select -from sqlalchemy.orm import Session - -from app.config import Settings -from app.models import ( - CrawlError, - CrawlRun, - CrawlRunEmployeeChange, - Employee, - EmployeeNewsLink, - EmployeePublication, - EmployeeSnapshot, - ParserSource, - ProfileTab, -) -from app.parser.collector import collect_profile_links -from app.parser.profile import parse_person_profile -from app.parser.profile_url import profile_key -from app.services.dataset_versions import get_or_create_current_version -from app.services.resource_cache import ResourceCache - -HEADERS = { - "User-Agent": "Mozilla/5.0 (compatible; MIEMEmployeesBot/0.1.0; +https://miem.hse.ru/)" -} - - +import gzip +import hashlib +import json +import re +import time +from datetime import datetime, timezone + +import requests +from sqlalchemy import inspect, select +from sqlalchemy.orm import Session + +from app.config import Settings +from app.models import ( + CrawlError, + CrawlRun, + CrawlRunEmployeeChange, + Employee, + EmployeeNewsLink, + EmployeePublication, + EmployeeProfileUrl, + EmployeeSnapshot, + ParserSource, + ProfileTab, +) +from app.parser.collector import collect_profile_links +from app.parser.profile import parse_person_profile +from app.parser.profile_url import profile_key +from app.services.dataset_versions import get_or_create_current_version +from app.services.resource_cache import ResourceCache + +HEADERS = { + "User-Agent": "Mozilla/5.0 (compatible; MIEMEmployeesBot/0.1.0; +https://miem.hse.ru/)" +} + + def run_crawl(db: Session, settings: Settings) -> CrawlRun: + source = _ensure_source(db, settings.source_url) + run = CrawlRun(source_url=source.source_url, status="running") + db.add(run) + db.commit() + db.refresh(run) + + found_keys: set[str] = set() + parsed_count = 0 + skipped_count = 0 + try: + with requests.Session() as session: + resource_cache = ResourceCache(db) + urls = collect_profile_links(session, source.source_url, HEADERS, settings.request_timeout) + if settings.crawl_limit: + urls = urls[: settings.crawl_limit] + run.found_count = len(urls) + db.commit() + + for url in urls: + key = profile_key(url) + if key: + found_keys.add(key) + try: + parsed = parse_person_profile( + session, + url, + HEADERS, + settings.request_timeout, + settings.parser_use_playwright, + resource_cache=resource_cache, + ) + if not parsed: + continue + employee, changed = _upsert_employee(db, run, parsed) + if employee.profile_key: + found_keys.add(employee.profile_key) + if changed: + parsed_count += 1 + else: + skipped_count += 1 + run.parsed_count = parsed_count + run.skipped_count = skipped_count + db.commit() + except Exception as exc: + run.error_count += 1 + db.add( + CrawlError( + crawl_run_id=run.id, + profile_url=url, + error_type=type(exc).__name__, + message=str(exc), + ) + ) + db.commit() + finally: + time.sleep(settings.request_delay_seconds) + + run.dismissed_count = _mark_dismissed( + db, + run, + found_keys, + session, + settings.request_timeout, + confirmation_runs=settings.dismissal_confirmation_runs, + max_auto_dismissals=settings.max_auto_dismissals_per_run, + ) + run.status = "completed" + get_or_create_current_version(db, crawl_run_id=run.id) + except Exception as exc: + run.status = "failed" + run.message = str(exc) + finally: + run.finished_at = datetime.now(timezone.utc) + db.commit() + db.refresh(run) + return run + + +def refresh_dismissed_status(db: Session, settings: Settings) -> CrawlRun: source = _ensure_source(db, settings.source_url) run = CrawlRun(source_url=source.source_url, status="running") db.add(run) db.commit() db.refresh(run) - found_keys: set[str] = set() - parsed_count = 0 - skipped_count = 0 try: + employees = db.scalars(select(Employee).where(Employee.status == "dismissed")).all() + run.found_count = len(employees) with requests.Session() as session: - resource_cache = ResourceCache(db) urls = collect_profile_links(session, source.source_url, HEADERS, settings.request_timeout) - if settings.crawl_limit: - urls = urls[: settings.crawl_limit] - run.found_count = len(urls) - db.commit() - - for url in urls: - key = profile_key(url) - if key: - found_keys.add(key) - try: - parsed = parse_person_profile( - session, - url, - HEADERS, - settings.request_timeout, - settings.parser_use_playwright, - resource_cache=resource_cache, - ) - if not parsed: - continue - _, changed = _upsert_employee(db, run, parsed) - if changed: - parsed_count += 1 - else: - skipped_count += 1 - run.parsed_count = parsed_count - run.skipped_count = skipped_count - db.commit() - except Exception as exc: - run.error_count += 1 - db.add( - CrawlError( - crawl_run_id=run.id, - profile_url=url, - error_type=type(exc).__name__, - message=str(exc), - ) - ) - db.commit() - finally: - time.sleep(settings.request_delay_seconds) - - run.dismissed_count = _mark_dismissed(db, run, found_keys, session, settings.request_timeout) - run.status = "completed" - get_or_create_current_version(db, crawl_run_id=run.id) - except Exception as exc: - run.status = "failed" - run.message = str(exc) - finally: - run.finished_at = datetime.now(timezone.utc) - db.commit() - db.refresh(run) - return run - - -def refresh_employee(db: Session, employee: Employee, settings: Settings) -> CrawlRun: - run = CrawlRun(source_url=employee.canonical_url, status="running", found_count=1) - db.add(run) - db.commit() - db.refresh(run) - - try: - with requests.Session() as session: - resource_cache = ResourceCache(db) - parsed = parse_person_profile( - session, - employee.canonical_url, - HEADERS, - settings.request_timeout, - settings.parser_use_playwright, - resource_cache=resource_cache, + source_keys = {key for url in urls if (key := profile_key(url))} + now = datetime.now(timezone.utc) + for employee in employees: + if employee.profile_key not in source_keys: + run.skipped_count += 1 + continue + employee.status = "active" + employee.dismissed_at = None + employee.last_seen_at = now + employee.profile_unavailable_streak = 0 + employee.last_profile_check_at = now + _record_employee_change( + db, + run, + employee, + "reactivated", + profile_available=True, + message="Сотрудник снова найден в исходном списке.", ) - if not parsed: - raise ValueError("Профиль не удалось распарсить.") - if _parsed_profile_key(parsed) != employee.profile_key: - raise ValueError("Распарсенный профиль не совпадает с обновляемым сотрудником.") - - _, changed = _upsert_employee(db, run, parsed) - if changed: - run.parsed_count = 1 - else: - run.skipped_count = 1 + run.parsed_count += 1 run.status = "completed" get_or_create_current_version(db, crawl_run_id=run.id) except Exception as exc: @@ -137,7 +153,7 @@ def refresh_employee(db: Session, employee: Employee, settings: Settings) -> Cra db.add( CrawlError( crawl_run_id=run.id, - profile_url=employee.canonical_url, + profile_url=source.source_url, error_type=type(exc).__name__, message=str(exc), ) @@ -149,392 +165,586 @@ def refresh_employee(db: Session, employee: Employee, settings: Settings) -> Cra return run -def _ensure_source(db: Session, source_url: str) -> ParserSource: - source = db.scalar(select(ParserSource).where(ParserSource.source_url == source_url)) - if source: - return source - source = ParserSource(source_url=source_url, enabled=True) - db.add(source) - db.commit() - db.refresh(source) - return source - - -def _parsed_profile_key(parsed: dict) -> str: - return f"{parsed.get('profile_type')}:{parsed.get('profile_id')}" - - -def _upsert_employee(db: Session, run: CrawlRun, parsed: dict) -> tuple[Employee, bool]: - html = parsed.pop("_html", None) - parsed.pop("_resource_manifest", None) - checksum = _checksum(parsed) - key = _parsed_profile_key(parsed) - employee = db.scalar(select(Employee).where(Employee.profile_key == key)) - now = datetime.now(timezone.utc) - if not employee: - employee = Employee( - profile_key=key, - profile_type=parsed.get("profile_type"), - profile_id=parsed.get("profile_id"), - canonical_url=parsed["source_url"], - first_seen_at=now, - ) - db.add(employee) - run.new_count += 1 - is_new = True - else: - is_new = False - - parser_version = parsed.get("parser_version") - changed = is_new or employee.current_checksum != checksum or employee.parser_version != parser_version - employee.full_name = parsed.get("full_name") - employee.status = "active" - employee.last_seen_at = now - employee.dismissed_at = None - employee.parser_version = parser_version - if changed: - employee.current_data = parsed - employee.current_checksum = checksum - db.flush() - - if is_new: - _record_employee_change( - db, - run, - employee, - "new", - profile_available=True, - message="Сотрудник впервые найден в источнике.", - ) - - if changed: - db.query(ProfileTab).filter(ProfileTab.employee_id == employee.id).delete() - for tab in parsed.get("tabs") or []: - db.add( - ProfileTab( - employee_id=employee.id, - title=tab.get("title") or "", - href=tab.get("href") or "", - data_index=tab.get("data_index"), - ) - ) - - db.add( - EmployeeSnapshot( - employee_id=employee.id, - crawl_run_id=run.id, - parsed_data=parsed, - html_snapshot=gzip.compress(html.encode("utf-8")) if html else None, - checksum=checksum, - parser_version=parser_version, - ) - ) - db.flush() - _try_sync_employee_publications(db, run, employee, parsed) - _try_sync_employee_news_links(db, run, employee, parsed) - return employee, changed - - -def _try_sync_employee_publications(db: Session, run: CrawlRun, employee: Employee, parsed: dict) -> None: - try: - if not _publication_payloads(parsed): - return - if not _employee_publications_table_exists(db): - return - with db.begin_nested(): - _sync_employee_publications(db, employee, parsed) - except Exception as exc: - db.add( - CrawlError( - crawl_run_id=run.id, - profile_url=employee.canonical_url, - error_type=type(exc).__name__, - message=f"Не удалось сохранить публикации сотрудника: {exc}", - ) - ) - - -def _employee_publications_table_exists(db: Session) -> bool: - return inspect(db.connection()).has_table(EmployeePublication.__tablename__) - - -def _sync_employee_publications(db: Session, employee: Employee, parsed: dict) -> None: - publications = _publication_payloads(parsed) - seen_hashes = set() - for publication in publications: - source_hash = _publication_hash(publication) - seen_hashes.add(source_hash) - publication_id = _clean_optional(publication.get("publication_id") or publication.get("id")) - existing = None - if publication_id: - existing = db.scalar( - select(EmployeePublication).where( - EmployeePublication.employee_id == employee.id, - EmployeePublication.publication_id == publication_id, - ) - ) - if not existing: - existing = db.scalar( - select(EmployeePublication).where( - EmployeePublication.employee_id == employee.id, - EmployeePublication.source_hash == source_hash, - ) - ) - if not existing: - existing = EmployeePublication(employee_id=employee.id, source_hash=source_hash, title=_publication_title(publication)) - db.add(existing) - _apply_publication(existing, publication, source_hash) - - if seen_hashes: - stale = db.scalars( - select(EmployeePublication).where( - EmployeePublication.employee_id == employee.id, - EmployeePublication.source_hash.not_in(seen_hashes), - ) - ).all() - for item in stale: - db.delete(item) - - -def _publication_payloads(parsed: dict) -> list[dict]: - publications = [] - for section in parsed.get("sections") or []: - if not isinstance(section, dict) or section.get("type") != "publications": - continue - for publication in section.get("publications") or []: - if isinstance(publication, dict): - publications.append(publication) - return publications - - -def _apply_publication(target: EmployeePublication, publication: dict, source_hash: str) -> None: - target.publication_id = _clean_optional(publication.get("publication_id") or publication.get("id")) - target.title = _publication_title(publication) - target.year = _int_or_none(publication.get("year")) - target.publication_type = _clean_optional(publication.get("publication_type") or publication.get("type")) - target.language = _clean_optional(publication.get("language")) - target.status = _int_or_none(publication.get("status")) - target.url = _clean_optional(publication.get("url")) - target.doi_url = _clean_optional(publication.get("doi_url")) - target.other_url = _clean_optional(publication.get("other_url")) - target.document_url = _clean_optional(publication.get("document_url")) - target.citation_text = _clean_optional(publication.get("citation_text") or publication.get("text")) - target.annotation = publication.get("annotation") if isinstance(publication.get("annotation"), dict) else None - target.description = publication.get("description") if isinstance(publication.get("description"), dict) else None - target.authors = publication.get("authors") if isinstance(publication.get("authors"), list) else None - target.raw_data = publication.get("raw_data") if isinstance(publication.get("raw_data"), dict) else publication - target.source_hash = source_hash - - -def _publication_hash(publication: dict) -> str: - return _payload_hash(publication.get("raw_data") if isinstance(publication.get("raw_data"), dict) else publication) - - -def _payload_hash(value: object) -> str: - payload = json.dumps(_stable_checksum_payload(value), ensure_ascii=False, sort_keys=True, separators=(",", ":"), default=str) - return hashlib.sha256(payload.encode("utf-8")).hexdigest() - - -def _publication_title(publication: dict) -> str: - return _clean_optional(publication.get("title") or publication.get("text") or publication.get("id")) or "Untitled publication" - - -def _clean_optional(value: object) -> str | None: - text = str(value or "").strip() - return text or None - - -def _int_or_none(value: object) -> int | None: - try: - return int(value) - except (TypeError, ValueError): - return None - - -def _try_sync_employee_news_links(db: Session, run: CrawlRun, employee: Employee, parsed: dict) -> None: - try: - if not _news_link_payloads(parsed): - return - if not _employee_news_links_table_exists(db): - return - with db.begin_nested(): - _sync_employee_news_links(db, employee, parsed) - except Exception as exc: - db.add( - CrawlError( - crawl_run_id=run.id, - profile_url=employee.canonical_url, - error_type=type(exc).__name__, - message=f"Не удалось сохранить новости сотрудника: {exc}", - ) - ) - - -def _employee_news_links_table_exists(db: Session) -> bool: - return inspect(db.connection()).has_table(EmployeeNewsLink.__tablename__) - - -def _sync_employee_news_links(db: Session, employee: Employee, parsed: dict) -> None: - news_links = _news_link_payloads(parsed) - seen_hashes = set() - for news_link in news_links: - source_hash = _news_link_hash(news_link) - seen_hashes.add(source_hash) - url = _clean_optional(news_link.get("url")) - existing = None - if url: - existing = db.scalar( - select(EmployeeNewsLink).where( - EmployeeNewsLink.employee_id == employee.id, - EmployeeNewsLink.url == url, - ) - ) - if not existing: - existing = db.scalar( - select(EmployeeNewsLink).where( - EmployeeNewsLink.employee_id == employee.id, - EmployeeNewsLink.source_hash == source_hash, - ) - ) - if not existing: - existing = EmployeeNewsLink(employee_id=employee.id, source_hash=source_hash, title=_news_link_title(news_link)) - db.add(existing) - _apply_news_link(existing, news_link, source_hash) - - if seen_hashes: - stale = db.scalars( - select(EmployeeNewsLink).where( - EmployeeNewsLink.employee_id == employee.id, - EmployeeNewsLink.source_hash.not_in(seen_hashes), - ) - ).all() - for item in stale: - db.delete(item) - - -def _news_link_payloads(parsed: dict) -> list[dict]: - news_links = [] - for section in parsed.get("sections") or []: - if not isinstance(section, dict) or section.get("type") != "news": - continue - for item in section.get("news_links") or []: - if isinstance(item, dict): - news_links.append(item) - return news_links - - -def _apply_news_link(target: EmployeeNewsLink, news_link: dict, source_hash: str) -> None: - target.title = _news_link_title(news_link) - target.url = _clean_optional(news_link.get("url")) - target.summary = _clean_optional(news_link.get("summary")) - target.published_at = _datetime_or_none(news_link.get("published_at")) - target.published_year = _int_or_none(news_link.get("published_year")) - target.raw_data = news_link.get("raw_data") if isinstance(news_link.get("raw_data"), dict) else news_link - target.source_hash = source_hash - - -def _news_link_hash(news_link: dict) -> str: - return _payload_hash(news_link.get("raw_data") if isinstance(news_link.get("raw_data"), dict) else news_link) - - -def _news_link_title(news_link: dict) -> str: - return _clean_optional(news_link.get("title") or news_link.get("url")) or "Untitled news" - - -def _datetime_or_none(value: object) -> datetime | None: - if isinstance(value, datetime): - return value - if not value: - return None - try: - parsed = datetime.fromisoformat(str(value).replace("Z", "+00:00")) - except ValueError: - return None - return parsed if parsed.tzinfo else parsed.replace(tzinfo=timezone.utc) - - -def _mark_dismissed(db: Session, run: CrawlRun, found_keys: set[str], session: requests.Session, timeout: int) -> int: - dismissed = 0 - active = db.scalars(select(Employee).where(Employee.status == "active")).all() - now = datetime.now(timezone.utc) - for employee in active: - if employee.profile_key in found_keys: - continue - profile_available = _profile_is_available(session, employee.canonical_url, timeout) - if profile_available: - _record_employee_change( - db, - run, - employee, - "missing_from_source", - profile_available=True, - message="Профиль доступен, но ссылка отсутствует в исходном списке.", - ) - continue - employee.status = "dismissed" - employee.dismissed_at = now - _record_employee_change( - db, - run, - employee, - "dismissed", - profile_available=False, - message="Сотрудник отсутствует в исходном списке, профиль не подтвердился как доступный.", - ) - dismissed += 1 - db.commit() - return dismissed - - -def _profile_is_available(session: requests.Session, url: str, timeout: int) -> bool: - try: - response = session.get(url, headers=HEADERS, timeout=timeout, allow_redirects=True) - return response.status_code < 400 - except requests.RequestException: - return False - - -def _record_employee_change( - db: Session, - run: CrawlRun, - employee: Employee, - change_type: str, - *, - profile_available: bool | None, - message: str, -) -> None: - db.add( - CrawlRunEmployeeChange( - crawl_run_id=run.id, - employee_id=employee.id, - profile_key=employee.profile_key, - profile_url=employee.canonical_url, - full_name=employee.full_name, - change_type=change_type, - profile_available=profile_available, - message=message, - ) - ) - - -def _checksum(data: dict) -> str: - payload = json.dumps(_stable_checksum_payload(data), ensure_ascii=False, sort_keys=True, separators=(",", ":")) - return hashlib.sha256(payload.encode("utf-8")).hexdigest() - - -def _stable_checksum_payload(value): - if isinstance(value, dict): - return {key: _stable_checksum_payload(item) for key, item in value.items()} - if isinstance(value, list): - return [_stable_checksum_payload(item) for item in value] - if isinstance(value, str): - return _normalize_date_dependent_experience(value) - return value - - -def _normalize_date_dependent_experience(value: str) -> str: - return re.sub( - r"(?i)(стаж(?:\s+работы)?(?:\s+в\s+ниу\s+вшэ|\s+в\s+вшэ)?\s*:?\s*)\d+\s*(?:год(?:а|ов)?|лет)", - r"\1", - value, - ) +def refresh_employee(db: Session, employee: Employee, settings: Settings) -> CrawlRun: + run = CrawlRun(source_url=employee.canonical_url, status="running", found_count=1) + db.add(run) + db.commit() + db.refresh(run) + + try: + with requests.Session() as session: + resource_cache = ResourceCache(db) + parsed = parse_person_profile( + session, + employee.canonical_url, + HEADERS, + settings.request_timeout, + settings.parser_use_playwright, + resource_cache=resource_cache, + ) + if not parsed: + raise ValueError("Профиль не удалось распарсить.") + if _parsed_profile_key(parsed) != employee.profile_key: + raise ValueError("Распарсенный профиль не совпадает с обновляемым сотрудником.") + + _, changed = _upsert_employee(db, run, parsed) + if changed: + run.parsed_count = 1 + else: + run.skipped_count = 1 + run.status = "completed" + get_or_create_current_version(db, crawl_run_id=run.id) + except Exception as exc: + run.status = "failed" + run.error_count = 1 + run.message = str(exc) + db.add( + CrawlError( + crawl_run_id=run.id, + profile_url=employee.canonical_url, + error_type=type(exc).__name__, + message=str(exc), + ) + ) + finally: + run.finished_at = datetime.now(timezone.utc) + db.commit() + db.refresh(run) + return run + + +def _ensure_source(db: Session, source_url: str) -> ParserSource: + source = db.scalar(select(ParserSource).where(ParserSource.source_url == source_url)) + if source: + return source + source = ParserSource(source_url=source_url, enabled=True) + db.add(source) + db.commit() + db.refresh(source) + return source + + +def _parsed_profile_key(parsed: dict) -> str: + return f"{parsed.get('profile_type')}:{parsed.get('profile_id')}" + + +def _upsert_employee(db: Session, run: CrawlRun, parsed: dict) -> tuple[Employee, bool]: + html = parsed.pop("_html", None) + parsed.pop("_resource_manifest", None) + checksum = _checksum(parsed) + key = _parsed_profile_key(parsed) + employee = db.scalar(select(Employee).where(Employee.profile_key == key)) + if not employee: + employee = _find_employee_with_moved_profile(db, parsed) + now = datetime.now(timezone.utc) + if not employee: + employee = Employee( + profile_key=key, + profile_type=parsed.get("profile_type"), + profile_id=parsed.get("profile_id"), + canonical_url=parsed["source_url"], + first_seen_at=now, + ) + db.add(employee) + run.new_count += 1 + is_new = True + else: + is_new = False + + parser_version = parsed.get("parser_version") + changed = is_new or employee.current_checksum != checksum or employee.parser_version != parser_version + previous_url = employee.canonical_url if employee.canonical_url != parsed["source_url"] else None + employee.profile_key = key + employee.profile_type = parsed.get("profile_type") + employee.profile_id = parsed.get("profile_id") + employee.canonical_url = parsed["source_url"] + employee.full_name = parsed.get("full_name") + employee.status = "active" + employee.last_seen_at = now + employee.dismissed_at = None + employee.profile_unavailable_streak = 0 + employee.last_profile_check_at = now + employee.parser_version = parser_version + if changed: + employee.current_data = parsed + employee.current_checksum = checksum + db.flush() + _sync_profile_url_history(db, employee, previous_url, employee.canonical_url, now) + + if is_new: + _record_employee_change( + db, + run, + employee, + "new", + profile_available=True, + message="Сотрудник впервые найден в источнике.", + ) + + if changed: + db.query(ProfileTab).filter(ProfileTab.employee_id == employee.id).delete() + for tab in parsed.get("tabs") or []: + db.add( + ProfileTab( + employee_id=employee.id, + title=tab.get("title") or "", + href=tab.get("href") or "", + data_index=tab.get("data_index"), + ) + ) + + db.add( + EmployeeSnapshot( + employee_id=employee.id, + crawl_run_id=run.id, + parsed_data=parsed, + html_snapshot=gzip.compress(html.encode("utf-8")) if html else None, + checksum=checksum, + parser_version=parser_version, + ) + ) + db.flush() + _try_sync_employee_publications(db, run, employee, parsed) + _try_sync_employee_news_links(db, run, employee, parsed) + return employee, changed + + +def _find_employee_with_moved_profile(db: Session, parsed: dict) -> Employee | None: + full_name = parsed.get("full_name") + if not full_name: + return None + candidates = db.scalars(select(Employee).where(Employee.full_name == full_name)).all() + if len(candidates) == 1: + return candidates[0] + + parsed_identity = _profile_identity_values(parsed) + if not parsed_identity: + return None + matches = [candidate for candidate in candidates if parsed_identity & _employee_identity_values(candidate)] + return matches[0] if len(matches) == 1 else None + + +def _profile_identity_values(profile: dict) -> set[str]: + if not isinstance(profile, dict): + return set() + values = set() + contacts = profile.get("contacts") or {} + if not isinstance(contacts, dict): + contacts = {} + for email in contacts.get("emails") or []: + normalized = str(email).strip().lower() + if normalized: + values.add(f"email:{normalized}") + for item in profile.get("external_ids") or []: + if not isinstance(item, dict): + continue + system = str(item.get("system") or "").strip().lower() + value = str(item.get("value") or "").strip().lower() + if system and value: + values.add(f"external:{system}:{value}") + return values + + +def _employee_identity_values(employee: Employee) -> set[str]: + return _profile_identity_values(employee.current_data or {}) + + +def _sync_profile_url_history( + db: Session, + employee: Employee, + previous_url: str | None, + current_url: str, + seen_at: datetime, +) -> None: + urls = {url for url in (previous_url, current_url) if url} + for url in urls: + history = db.scalar( + select(EmployeeProfileUrl).where( + EmployeeProfileUrl.employee_id == employee.id, + EmployeeProfileUrl.url == url, + ) + ) + if history: + history.last_seen_at = seen_at + else: + db.add( + EmployeeProfileUrl( + employee_id=employee.id, + url=url, + first_seen_at=seen_at, + last_seen_at=seen_at, + ) + ) + + +def _try_sync_employee_publications(db: Session, run: CrawlRun, employee: Employee, parsed: dict) -> None: + try: + if not _publication_payloads(parsed): + return + if not _employee_publications_table_exists(db): + return + with db.begin_nested(): + _sync_employee_publications(db, employee, parsed) + except Exception as exc: + db.add( + CrawlError( + crawl_run_id=run.id, + profile_url=employee.canonical_url, + error_type=type(exc).__name__, + message=f"Не удалось сохранить публикации сотрудника: {exc}", + ) + ) + + +def _employee_publications_table_exists(db: Session) -> bool: + return inspect(db.connection()).has_table(EmployeePublication.__tablename__) + + +def _sync_employee_publications(db: Session, employee: Employee, parsed: dict) -> None: + publications = _publication_payloads(parsed) + seen_hashes = set() + for publication in publications: + source_hash = _publication_hash(publication) + seen_hashes.add(source_hash) + publication_id = _clean_optional(publication.get("publication_id") or publication.get("id")) + existing = None + if publication_id: + existing = db.scalar( + select(EmployeePublication).where( + EmployeePublication.employee_id == employee.id, + EmployeePublication.publication_id == publication_id, + ) + ) + if not existing: + existing = db.scalar( + select(EmployeePublication).where( + EmployeePublication.employee_id == employee.id, + EmployeePublication.source_hash == source_hash, + ) + ) + if not existing: + existing = EmployeePublication(employee_id=employee.id, source_hash=source_hash, title=_publication_title(publication)) + db.add(existing) + _apply_publication(existing, publication, source_hash) + + if seen_hashes: + stale = db.scalars( + select(EmployeePublication).where( + EmployeePublication.employee_id == employee.id, + EmployeePublication.source_hash.not_in(seen_hashes), + ) + ).all() + for item in stale: + db.delete(item) + + +def _publication_payloads(parsed: dict) -> list[dict]: + publications = [] + for section in parsed.get("sections") or []: + if not isinstance(section, dict) or section.get("type") != "publications": + continue + for publication in section.get("publications") or []: + if isinstance(publication, dict): + publications.append(publication) + return publications + + +def _apply_publication(target: EmployeePublication, publication: dict, source_hash: str) -> None: + target.publication_id = _clean_optional(publication.get("publication_id") or publication.get("id")) + target.title = _publication_title(publication) + target.year = _int_or_none(publication.get("year")) + target.publication_type = _clean_optional(publication.get("publication_type") or publication.get("type")) + target.language = _clean_optional(publication.get("language")) + target.status = _int_or_none(publication.get("status")) + target.url = _clean_optional(publication.get("url")) + target.doi_url = _clean_optional(publication.get("doi_url")) + target.other_url = _clean_optional(publication.get("other_url")) + target.document_url = _clean_optional(publication.get("document_url")) + target.citation_text = _clean_optional(publication.get("citation_text") or publication.get("text")) + target.annotation = publication.get("annotation") if isinstance(publication.get("annotation"), dict) else None + target.description = publication.get("description") if isinstance(publication.get("description"), dict) else None + target.authors = publication.get("authors") if isinstance(publication.get("authors"), list) else None + target.raw_data = publication.get("raw_data") if isinstance(publication.get("raw_data"), dict) else publication + target.source_hash = source_hash + + +def _publication_hash(publication: dict) -> str: + return _payload_hash(publication.get("raw_data") if isinstance(publication.get("raw_data"), dict) else publication) + + +def _payload_hash(value: object) -> str: + payload = json.dumps(_stable_checksum_payload(value), ensure_ascii=False, sort_keys=True, separators=(",", ":"), default=str) + return hashlib.sha256(payload.encode("utf-8")).hexdigest() + + +def _publication_title(publication: dict) -> str: + return _clean_optional(publication.get("title") or publication.get("text") or publication.get("id")) or "Untitled publication" + + +def _clean_optional(value: object) -> str | None: + text = str(value or "").strip() + return text or None + + +def _int_or_none(value: object) -> int | None: + try: + return int(value) + except (TypeError, ValueError): + return None + + +def _try_sync_employee_news_links(db: Session, run: CrawlRun, employee: Employee, parsed: dict) -> None: + try: + if not _news_link_payloads(parsed): + return + if not _employee_news_links_table_exists(db): + return + with db.begin_nested(): + _sync_employee_news_links(db, employee, parsed) + except Exception as exc: + db.add( + CrawlError( + crawl_run_id=run.id, + profile_url=employee.canonical_url, + error_type=type(exc).__name__, + message=f"Не удалось сохранить новости сотрудника: {exc}", + ) + ) + + +def _employee_news_links_table_exists(db: Session) -> bool: + return inspect(db.connection()).has_table(EmployeeNewsLink.__tablename__) + + +def _sync_employee_news_links(db: Session, employee: Employee, parsed: dict) -> None: + news_links = _news_link_payloads(parsed) + seen_hashes = set() + for news_link in news_links: + source_hash = _news_link_hash(news_link) + seen_hashes.add(source_hash) + url = _clean_optional(news_link.get("url")) + existing = None + if url: + existing = db.scalar( + select(EmployeeNewsLink).where( + EmployeeNewsLink.employee_id == employee.id, + EmployeeNewsLink.url == url, + ) + ) + if not existing: + existing = db.scalar( + select(EmployeeNewsLink).where( + EmployeeNewsLink.employee_id == employee.id, + EmployeeNewsLink.source_hash == source_hash, + ) + ) + if not existing: + existing = EmployeeNewsLink(employee_id=employee.id, source_hash=source_hash, title=_news_link_title(news_link)) + db.add(existing) + _apply_news_link(existing, news_link, source_hash) + + if seen_hashes: + stale = db.scalars( + select(EmployeeNewsLink).where( + EmployeeNewsLink.employee_id == employee.id, + EmployeeNewsLink.source_hash.not_in(seen_hashes), + ) + ).all() + for item in stale: + db.delete(item) + + +def _news_link_payloads(parsed: dict) -> list[dict]: + news_links = [] + for section in parsed.get("sections") or []: + if not isinstance(section, dict) or section.get("type") != "news": + continue + for item in section.get("news_links") or []: + if isinstance(item, dict): + news_links.append(item) + return news_links + + +def _apply_news_link(target: EmployeeNewsLink, news_link: dict, source_hash: str) -> None: + target.title = _news_link_title(news_link) + target.url = _clean_optional(news_link.get("url")) + target.summary = _clean_optional(news_link.get("summary")) + target.published_at = _datetime_or_none(news_link.get("published_at")) + target.published_year = _int_or_none(news_link.get("published_year")) + target.raw_data = news_link.get("raw_data") if isinstance(news_link.get("raw_data"), dict) else news_link + target.source_hash = source_hash + + +def _news_link_hash(news_link: dict) -> str: + return _payload_hash(news_link.get("raw_data") if isinstance(news_link.get("raw_data"), dict) else news_link) + + +def _news_link_title(news_link: dict) -> str: + return _clean_optional(news_link.get("title") or news_link.get("url")) or "Untitled news" + + +def _datetime_or_none(value: object) -> datetime | None: + if isinstance(value, datetime): + return value + if not value: + return None + try: + parsed = datetime.fromisoformat(str(value).replace("Z", "+00:00")) + except ValueError: + return None + return parsed if parsed.tzinfo else parsed.replace(tzinfo=timezone.utc) + + +def _mark_dismissed( + db: Session, + run: CrawlRun, + found_keys: set[str], + session: requests.Session, + timeout: int, + *, + confirmation_runs: int = 3, + max_auto_dismissals: int | None = 25, +) -> int: + dismissed = 0 + candidates = db.scalars( + select(Employee).where(Employee.status.in_(("active", "verification_required"))) + ).all() + now = datetime.now(timezone.utc) + unavailable = [] + for employee in candidates: + if employee.profile_key in found_keys: + continue + profile_available = _profile_check(session, employee.canonical_url, timeout) + employee.last_profile_check_at = now + if profile_available is None: + db.add( + CrawlError( + crawl_run_id=run.id, + profile_url=employee.canonical_url, + error_type="ProfileAvailabilityCheckError", + message="Не удалось надёжно проверить доступность профиля; статус сотрудника не изменён.", + ) + ) + continue + if profile_available: + employee.profile_unavailable_streak = 0 + if employee.status == "verification_required": + employee.status = "active" + _record_employee_change( + db, + run, + employee, + "missing_from_source", + profile_available=True, + message="Профиль доступен, но ссылка отсутствует в исходном списке.", + ) + continue + next_streak = employee.profile_unavailable_streak + 1 + unavailable.append((employee, next_streak)) + + dismissal_blocked = bool( + max_auto_dismissals is not None and len(unavailable) > max_auto_dismissals + ) + if dismissal_blocked: + run.message = ( + f"Автоматическое увольнение приостановлено: {len(unavailable)} профилей " + f"одновременно не подтвердились (лимит {max_auto_dismissals})." + ) + + for employee, next_streak in unavailable: + employee.profile_unavailable_streak = next_streak + if next_streak < confirmation_runs or dismissal_blocked: + employee.status = "verification_required" + _record_employee_change( + db, + run, + employee, + "verification_required", + profile_available=False, + message=( + "Профиль не подтвердился. Автоматическое увольнение отложено до " + f"{confirmation_runs} последовательных проверок." + if not dismissal_blocked + else "Автоматическое увольнение отложено из-за массовой ошибки проверки профилей." + ), + ) + continue + + employee.status = "dismissed" + employee.dismissed_at = now + _record_employee_change( + db, + run, + employee, + "dismissed", + profile_available=False, + message=( + "Сотрудник отсутствует в исходном списке, профиль не подтвердился " + f"{confirmation_runs} раза подряд." + ), + ) + dismissed += 1 + db.commit() + return dismissed + + +def _profile_is_available(session: requests.Session, url: str, timeout: int) -> bool: + return _profile_check(session, url, timeout) is True + + +def _profile_check(session: requests.Session, url: str, timeout: int) -> bool | None: + try: + response = session.get(url, headers=HEADERS, timeout=timeout, allow_redirects=True) + if response.status_code < 400: + return True + if response.status_code in {404, 410}: + return False + return None + except requests.RequestException: + return None + + +def _record_employee_change( + db: Session, + run: CrawlRun, + employee: Employee, + change_type: str, + *, + profile_available: bool | None, + message: str, +) -> None: + db.add( + CrawlRunEmployeeChange( + crawl_run_id=run.id, + employee_id=employee.id, + profile_key=employee.profile_key, + profile_url=employee.canonical_url, + full_name=employee.full_name, + change_type=change_type, + profile_available=profile_available, + message=message, + ) + ) + + +def _checksum(data: dict) -> str: + payload = json.dumps(_stable_checksum_payload(data), ensure_ascii=False, sort_keys=True, separators=(",", ":")) + return hashlib.sha256(payload.encode("utf-8")).hexdigest() + + +def _stable_checksum_payload(value): + if isinstance(value, dict): + return {key: _stable_checksum_payload(item) for key, item in value.items()} + if isinstance(value, list): + return [_stable_checksum_payload(item) for item in value] + if isinstance(value, str): + return _normalize_date_dependent_experience(value) + return value + + +def _normalize_date_dependent_experience(value: str) -> str: + return re.sub( + r"(?i)(стаж(?:\s+работы)?(?:\s+в\s+ниу\s+вшэ|\s+в\s+вшэ)?\s*:?\s*)\d+\s*(?:год(?:а|ов)?|лет)", + r"\1", + value, + ) diff --git a/app/static/admin.css b/app/static/admin.css index 0a0f4b3..51a8dd8 100644 --- a/app/static/admin.css +++ b/app/static/admin.css @@ -1,426 +1,431 @@ -.admin { - margin: 0; - min-height: 100vh; - display: flex; - flex-direction: column; - color: #1f2937; - background: #f6f7f9; - font-family: Arial, sans-serif; -} - -.admin__header { - display: flex; - align-items: center; - justify-content: space-between; - gap: 24px; - padding: 18px 32px; - background: #ffffff; - border-bottom: 1px solid #d9dee7; -} - -.admin__brand { - margin: 0; - font-size: 20px; -} - -.admin__brand-link { - color: inherit; - text-decoration: none; -} - -.admin__nav { - display: flex; - align-items: center; - gap: 14px; -} - -.admin__link { - color: #0f766e; - text-decoration: none; - font-weight: 700; -} - -.admin__main { - flex: 1; - width: min(1180px, calc(100% - 32px)); - margin: 28px auto; -} - -.admin__footer { - padding: 20px 32px; - color: #6b7280; - border-top: 1px solid #d9dee7; - background: #ffffff; -} - -.admin__grid { - display: grid; - grid-template-columns: repeat(auto-fit, minmax(180px, 1fr)); - gap: 16px; -} - -.metric { - display: block; - padding: 18px; - background: #ffffff; - border: 1px solid #d9dee7; - border-radius: 8px; -} - -.metric--link { - color: inherit; - text-decoration: none; -} - -.metric--link:hover { - border-color: #0f766e; -} - -.metric__label { - display: block; - color: #6b7280; - font-size: 13px; -} - -.metric__value { - display: block; - margin-top: 8px; - font-size: 28px; - font-weight: 700; -} - -.panel { - margin-top: 22px; - padding: 20px; - background: #ffffff; - border: 1px solid #d9dee7; - border-radius: 8px; -} - -.panel__title { - margin: 0 0 16px; - font-size: 18px; -} - -.table { - width: 100%; - border-collapse: collapse; -} - -.table__row { - cursor: pointer; -} - -.table__row:hover { - background: #f0fdfa; -} - -.table__cell, -.table__head { - padding: 10px 8px; - border-bottom: 1px solid #e5e7eb; - text-align: left; - vertical-align: top; -} - -.badge { - display: inline-block; - padding: 3px 8px; - border-radius: 999px; - background: #e0f2fe; - color: #075985; - font-size: 12px; -} - -.badge--dismissed { - background: #fee2e2; - color: #991b1b; -} - -.form { - display: grid; - gap: 12px; - max-width: 380px; -} - -.form__label { - display: grid; - gap: 6px; - font-weight: 700; -} - -.form__input, -.form__select { - padding: 10px 12px; - border: 1px solid #cbd5e1; - border-radius: 6px; -} - -.button { - padding: 10px 14px; - border: 0; - border-radius: 6px; - color: #ffffff; - background: #0f766e; - font-weight: 700; - cursor: pointer; -} - -.button--ghost { - color: #0f766e; - background: transparent; -} - -.button--compact { - padding: 8px 12px; -} - -.code { - overflow-x: auto; - padding: 14px; - background: #111827; - color: #f9fafb; - border-radius: 8px; - white-space: pre-wrap; -} - -.employee-card { - display: grid; - gap: 18px; -} - -.employee-card__header { - display: flex; - align-items: flex-start; - justify-content: space-between; - gap: 18px; - padding: 22px; - background: #ffffff; - border: 1px solid #d9dee7; - border-radius: 8px; -} - -.employee-card__identity { - display: grid; - gap: 10px; -} - -.employee-card__actions { - display: grid; - justify-items: end; - gap: 10px; -} - -.employee-card__title { - margin: 0; - font-size: 24px; -} - -.employee-card__notice { - margin: 0; - padding: 12px 14px; - border-radius: 8px; - font-weight: 700; -} - -.employee-card__notice--success { - color: #065f46; - background: #d1fae5; -} - -.employee-card__notice--error { - color: #991b1b; - background: #fee2e2; -} - -.employee-card__section { - padding: 20px; - background: #ffffff; - border: 1px solid #d9dee7; - border-radius: 8px; -} - -.employee-card__meta { - display: grid; - grid-template-columns: repeat(auto-fit, minmax(240px, 1fr)); - gap: 14px; - margin: 0; -} - -.employee-card__meta-item { - min-width: 0; -} - -.employee-card__meta-item--wide { - grid-column: 1 / -1; -} - -.employee-card__meta-label { - margin-bottom: 5px; - color: #6b7280; - font-size: 12px; - font-weight: 700; - text-transform: uppercase; -} - -.employee-card__meta-value { - margin: 0; - color: #1f2937; - line-height: 1.45; -} - -.employee-card__list { - display: grid; - gap: 8px; - margin: 0; - padding-left: 18px; -} - -.employee-card__list-item { - line-height: 1.45; -} - -.employee-card__sections { - display: grid; - gap: 14px; -} - -.employee-section { - padding: 16px; - background: #f9fafb; - border: 1px solid #e5e7eb; - border-radius: 8px; -} - -.employee-section__header { - display: flex; - align-items: flex-start; - justify-content: space-between; - gap: 12px; - margin-bottom: 12px; -} - -.employee-section__title { - margin: 0; - font-size: 17px; -} - -.employee-section__type { - flex: 0 0 auto; - padding: 3px 8px; - color: #475569; - background: #e2e8f0; - border-radius: 999px; - font-size: 12px; -} - -.employee-section__note { - margin: 0 0 10px; - color: #4b5563; - font-weight: 700; -} - -.employee-section__text { - margin: 0 0 10px; - line-height: 1.55; -} - -.employee-section__meta { - display: flex; - flex-wrap: wrap; - gap: 8px 12px; - color: #4b5563; - font-size: 13px; -} - -.employee-section__meta-item { - line-height: 1.4; -} - -.employee-section__table-wrap { - overflow-x: auto; -} - -.employee-section__table { - width: 100%; - border-collapse: collapse; - background: #ffffff; -} - -.employee-section__head, -.employee-section__cell { - padding: 10px; - border-bottom: 1px solid #e5e7eb; - text-align: left; - vertical-align: top; -} - -.employee-section__head { - color: #374151; - background: #f3f4f6; - font-size: 13px; -} - -.employee-section__links { - display: flex; - flex-wrap: wrap; - gap: 8px; - margin-top: 12px; -} - -.employee-section__link { - padding: 5px 9px; - color: #0f766e; - background: #ccfbf1; - border-radius: 999px; - font-size: 12px; - font-weight: 700; - text-decoration: none; -} - -.stats-strip { - display: grid; - grid-template-columns: repeat(auto-fit, minmax(220px, 1fr)); - gap: 14px; - margin-top: 16px; -} - -.stats-strip__item { - display: block; - padding: 14px 16px; - background: #ffffff; - border: 1px solid #d9dee7; - border-radius: 8px; -} - -.stats-strip__item--link { - color: inherit; - text-decoration: none; -} - -.stats-strip__item--link:hover { - border-color: #0f766e; -} - -.stats-strip__label { - display: block; - color: #6b7280; - font-size: 12px; - text-transform: uppercase; -} - -.stats-strip__value { - display: block; - margin-top: 6px; - color: #1f2937; - font-weight: 700; -} - -.progress-panel { - display: grid; - gap: 12px; -} - +.admin { + margin: 0; + min-height: 100vh; + display: flex; + flex-direction: column; + color: #1f2937; + background: #f6f7f9; + font-family: Arial, sans-serif; +} + +.admin__header { + display: flex; + align-items: center; + justify-content: space-between; + gap: 24px; + padding: 18px 32px; + background: #ffffff; + border-bottom: 1px solid #d9dee7; +} + +.admin__brand { + margin: 0; + font-size: 20px; +} + +.admin__brand-link { + color: inherit; + text-decoration: none; +} + +.admin__nav { + display: flex; + align-items: center; + gap: 14px; +} + +.admin__link { + color: #0f766e; + text-decoration: none; + font-weight: 700; +} + +.admin__main { + flex: 1; + width: min(1180px, calc(100% - 32px)); + margin: 28px auto; +} + +.admin__footer { + padding: 20px 32px; + color: #6b7280; + border-top: 1px solid #d9dee7; + background: #ffffff; +} + +.admin__grid { + display: grid; + grid-template-columns: repeat(auto-fit, minmax(180px, 1fr)); + gap: 16px; +} + +.metric { + display: block; + padding: 18px; + background: #ffffff; + border: 1px solid #d9dee7; + border-radius: 8px; +} + +.metric--link { + color: inherit; + text-decoration: none; +} + +.metric--link:hover { + border-color: #0f766e; +} + +.metric__label { + display: block; + color: #6b7280; + font-size: 13px; +} + +.metric__value { + display: block; + margin-top: 8px; + font-size: 28px; + font-weight: 700; +} + +.panel { + margin-top: 22px; + padding: 20px; + background: #ffffff; + border: 1px solid #d9dee7; + border-radius: 8px; +} + +.panel__title { + margin: 0 0 16px; + font-size: 18px; +} + +.table { + width: 100%; + border-collapse: collapse; +} + +.table__row { + cursor: pointer; +} + +.table__row:hover { + background: #f0fdfa; +} + +.table__cell, +.table__head { + padding: 10px 8px; + border-bottom: 1px solid #e5e7eb; + text-align: left; + vertical-align: top; +} + +.badge { + display: inline-block; + padding: 3px 8px; + border-radius: 999px; + background: #e0f2fe; + color: #075985; + font-size: 12px; +} + +.badge--dismissed { + background: #fee2e2; + color: #991b1b; +} + +.badge--verification { + background: #fef3c7; + color: #92400e; +} + +.form { + display: grid; + gap: 12px; + max-width: 380px; +} + +.form__label { + display: grid; + gap: 6px; + font-weight: 700; +} + +.form__input, +.form__select { + padding: 10px 12px; + border: 1px solid #cbd5e1; + border-radius: 6px; +} + +.button { + padding: 10px 14px; + border: 0; + border-radius: 6px; + color: #ffffff; + background: #0f766e; + font-weight: 700; + cursor: pointer; +} + +.button--ghost { + color: #0f766e; + background: transparent; +} + +.button--compact { + padding: 8px 12px; +} + +.code { + overflow-x: auto; + padding: 14px; + background: #111827; + color: #f9fafb; + border-radius: 8px; + white-space: pre-wrap; +} + +.employee-card { + display: grid; + gap: 18px; +} + +.employee-card__header { + display: flex; + align-items: flex-start; + justify-content: space-between; + gap: 18px; + padding: 22px; + background: #ffffff; + border: 1px solid #d9dee7; + border-radius: 8px; +} + +.employee-card__identity { + display: grid; + gap: 10px; +} + +.employee-card__actions { + display: grid; + justify-items: end; + gap: 10px; +} + +.employee-card__title { + margin: 0; + font-size: 24px; +} + +.employee-card__notice { + margin: 0; + padding: 12px 14px; + border-radius: 8px; + font-weight: 700; +} + +.employee-card__notice--success { + color: #065f46; + background: #d1fae5; +} + +.employee-card__notice--error { + color: #991b1b; + background: #fee2e2; +} + +.employee-card__section { + padding: 20px; + background: #ffffff; + border: 1px solid #d9dee7; + border-radius: 8px; +} + +.employee-card__meta { + display: grid; + grid-template-columns: repeat(auto-fit, minmax(240px, 1fr)); + gap: 14px; + margin: 0; +} + +.employee-card__meta-item { + min-width: 0; +} + +.employee-card__meta-item--wide { + grid-column: 1 / -1; +} + +.employee-card__meta-label { + margin-bottom: 5px; + color: #6b7280; + font-size: 12px; + font-weight: 700; + text-transform: uppercase; +} + +.employee-card__meta-value { + margin: 0; + color: #1f2937; + line-height: 1.45; +} + +.employee-card__list { + display: grid; + gap: 8px; + margin: 0; + padding-left: 18px; +} + +.employee-card__list-item { + line-height: 1.45; +} + +.employee-card__sections { + display: grid; + gap: 14px; +} + +.employee-section { + padding: 16px; + background: #f9fafb; + border: 1px solid #e5e7eb; + border-radius: 8px; +} + +.employee-section__header { + display: flex; + align-items: flex-start; + justify-content: space-between; + gap: 12px; + margin-bottom: 12px; +} + +.employee-section__title { + margin: 0; + font-size: 17px; +} + +.employee-section__type { + flex: 0 0 auto; + padding: 3px 8px; + color: #475569; + background: #e2e8f0; + border-radius: 999px; + font-size: 12px; +} + +.employee-section__note { + margin: 0 0 10px; + color: #4b5563; + font-weight: 700; +} + +.employee-section__text { + margin: 0 0 10px; + line-height: 1.55; +} + +.employee-section__meta { + display: flex; + flex-wrap: wrap; + gap: 8px 12px; + color: #4b5563; + font-size: 13px; +} + +.employee-section__meta-item { + line-height: 1.4; +} + +.employee-section__table-wrap { + overflow-x: auto; +} + +.employee-section__table { + width: 100%; + border-collapse: collapse; + background: #ffffff; +} + +.employee-section__head, +.employee-section__cell { + padding: 10px; + border-bottom: 1px solid #e5e7eb; + text-align: left; + vertical-align: top; +} + +.employee-section__head { + color: #374151; + background: #f3f4f6; + font-size: 13px; +} + +.employee-section__links { + display: flex; + flex-wrap: wrap; + gap: 8px; + margin-top: 12px; +} + +.employee-section__link { + padding: 5px 9px; + color: #0f766e; + background: #ccfbf1; + border-radius: 999px; + font-size: 12px; + font-weight: 700; + text-decoration: none; +} + +.stats-strip { + display: grid; + grid-template-columns: repeat(auto-fit, minmax(220px, 1fr)); + gap: 14px; + margin-top: 16px; +} + +.stats-strip__item { + display: block; + padding: 14px 16px; + background: #ffffff; + border: 1px solid #d9dee7; + border-radius: 8px; +} + +.stats-strip__item--link { + color: inherit; + text-decoration: none; +} + +.stats-strip__item--link:hover { + border-color: #0f766e; +} + +.stats-strip__label { + display: block; + color: #6b7280; + font-size: 12px; + text-transform: uppercase; +} + +.stats-strip__value { + display: block; + margin-top: 6px; + color: #1f2937; + font-weight: 700; +} + +.progress-panel { + display: grid; + gap: 12px; +} + .progress-panel__header { display: flex; align-items: center; @@ -428,222 +433,233 @@ gap: 16px; } -.progress-panel__body { - display: grid; - gap: 10px; -} - -.progress-panel__meta { +.progress-panel__actions { display: flex; flex-wrap: wrap; - gap: 12px; - color: #4b5563; - font-size: 14px; -} - -.progress-panel__percent { - color: #0f766e; - font-weight: 700; -} - -.progress-panel__empty { - margin: 0; - color: #6b7280; -} - -.progress-bar { - height: 12px; - overflow: hidden; - background: #e5e7eb; - border-radius: 999px; -} - -.progress-bar__fill { - height: 100%; - width: 0; - background: #0f766e; - transition: width 0.25s ease; -} - -.directory { - display: grid; - gap: 18px; -} - -.directory__header { - display: flex; - align-items: end; - justify-content: space-between; - gap: 16px; -} - -.directory__title { - margin: 0; - font-size: 24px; -} - -.directory__summary { - margin: 6px 0 0; - color: #6b7280; -} - -.directory__filters { - display: grid; - grid-template-columns: minmax(220px, 1.7fr) repeat(6, minmax(120px, 1fr)); - gap: 10px; - padding: 16px; - background: #ffffff; - border: 1px solid #d9dee7; - border-radius: 8px; -} - -.directory__input { - min-width: 0; - padding: 10px 12px; - border: 1px solid #cbd5e1; - border-radius: 6px; -} - -.directory__table-wrap { - overflow-x: auto; - background: #ffffff; - border: 1px solid #d9dee7; - border-radius: 8px; -} - -.directory__pagination { - display: flex; - align-items: center; - justify-content: center; - gap: 16px; -} - -.directory__page { - color: #4b5563; - font-weight: 700; -} - -.directory-table { - width: 100%; - min-width: 1120px; - border-collapse: collapse; -} - -.directory-table__head { - padding: 12px 10px; - color: #374151; - background: #f9fafb; - border-bottom: 1px solid #e5e7eb; - font-size: 13px; - text-align: left; - white-space: nowrap; -} - -.directory-table__cell { - max-width: 280px; - padding: 12px 10px; - border-bottom: 1px solid #e5e7eb; - vertical-align: top; -} - -.directory-table__row { - cursor: pointer; -} - -.directory-table__row:hover { - background: #f0fdfa; -} - -.directory-table__empty { - padding: 28px; - color: #6b7280; - text-align: center; -} - -.directory-table__cell--hidden, -.directory-table__head--hidden { - display: none; -} - -.columns-modal { - position: fixed; - inset: 0; - z-index: 50; - display: grid; - place-items: center; - padding: 20px; -} - -.columns-modal[hidden] { - display: none; -} - -.columns-modal__backdrop { - position: absolute; - inset: 0; - background: rgba(17, 24, 39, 0.54); -} - -.columns-modal__panel { - position: relative; - width: min(620px, 100%); - max-height: min(720px, calc(100vh - 40px)); - overflow: auto; - padding: 20px; - background: #ffffff; - border-radius: 8px; - box-shadow: 0 24px 80px rgba(15, 23, 42, 0.22); -} - -.columns-modal__header { - display: flex; - align-items: center; - justify-content: space-between; - gap: 12px; -} - -.columns-modal__title { - margin: 0; - font-size: 18px; -} - -.columns-modal__grid { - display: grid; - grid-template-columns: repeat(auto-fit, minmax(180px, 1fr)); - gap: 10px; - margin-top: 18px; -} - -.columns-modal__option { - display: flex; - align-items: center; gap: 8px; - padding: 10px; - background: #f9fafb; - border: 1px solid #e5e7eb; - border-radius: 6px; } -.columns-modal__checkbox { - width: 16px; - height: 16px; +.button--secondary { + color: #0f766e; + background: #ccfbf1; } -@media (max-width: 920px) { - .directory__filters { - grid-template-columns: 1fr 1fr; - } - - .progress-panel__header, - .directory__header, - .employee-card__header { - align-items: stretch; - flex-direction: column; - } -} - -@media (max-width: 620px) { - .directory__filters { - grid-template-columns: 1fr; - } -} +.progress-panel__body { + display: grid; + gap: 10px; +} + +.progress-panel__meta { + display: flex; + flex-wrap: wrap; + gap: 12px; + color: #4b5563; + font-size: 14px; +} + +.progress-panel__percent { + color: #0f766e; + font-weight: 700; +} + +.progress-panel__empty { + margin: 0; + color: #6b7280; +} + +.progress-bar { + height: 12px; + overflow: hidden; + background: #e5e7eb; + border-radius: 999px; +} + +.progress-bar__fill { + height: 100%; + width: 0; + background: #0f766e; + transition: width 0.25s ease; +} + +.directory { + display: grid; + gap: 18px; +} + +.directory__header { + display: flex; + align-items: end; + justify-content: space-between; + gap: 16px; +} + +.directory__title { + margin: 0; + font-size: 24px; +} + +.directory__summary { + margin: 6px 0 0; + color: #6b7280; +} + +.directory__filters { + display: grid; + grid-template-columns: minmax(220px, 1.7fr) repeat(6, minmax(120px, 1fr)); + gap: 10px; + padding: 16px; + background: #ffffff; + border: 1px solid #d9dee7; + border-radius: 8px; +} + +.directory__input { + min-width: 0; + padding: 10px 12px; + border: 1px solid #cbd5e1; + border-radius: 6px; +} + +.directory__table-wrap { + overflow-x: auto; + background: #ffffff; + border: 1px solid #d9dee7; + border-radius: 8px; +} + +.directory__pagination { + display: flex; + align-items: center; + justify-content: center; + gap: 16px; +} + +.directory__page { + color: #4b5563; + font-weight: 700; +} + +.directory-table { + width: 100%; + min-width: 1120px; + border-collapse: collapse; +} + +.directory-table__head { + padding: 12px 10px; + color: #374151; + background: #f9fafb; + border-bottom: 1px solid #e5e7eb; + font-size: 13px; + text-align: left; + white-space: nowrap; +} + +.directory-table__cell { + max-width: 280px; + padding: 12px 10px; + border-bottom: 1px solid #e5e7eb; + vertical-align: top; +} + +.directory-table__row { + cursor: pointer; +} + +.directory-table__row:hover { + background: #f0fdfa; +} + +.directory-table__empty { + padding: 28px; + color: #6b7280; + text-align: center; +} + +.directory-table__cell--hidden, +.directory-table__head--hidden { + display: none; +} + +.columns-modal { + position: fixed; + inset: 0; + z-index: 50; + display: grid; + place-items: center; + padding: 20px; +} + +.columns-modal[hidden] { + display: none; +} + +.columns-modal__backdrop { + position: absolute; + inset: 0; + background: rgba(17, 24, 39, 0.54); +} + +.columns-modal__panel { + position: relative; + width: min(620px, 100%); + max-height: min(720px, calc(100vh - 40px)); + overflow: auto; + padding: 20px; + background: #ffffff; + border-radius: 8px; + box-shadow: 0 24px 80px rgba(15, 23, 42, 0.22); +} + +.columns-modal__header { + display: flex; + align-items: center; + justify-content: space-between; + gap: 12px; +} + +.columns-modal__title { + margin: 0; + font-size: 18px; +} + +.columns-modal__grid { + display: grid; + grid-template-columns: repeat(auto-fit, minmax(180px, 1fr)); + gap: 10px; + margin-top: 18px; +} + +.columns-modal__option { + display: flex; + align-items: center; + gap: 8px; + padding: 10px; + background: #f9fafb; + border: 1px solid #e5e7eb; + border-radius: 6px; +} + +.columns-modal__checkbox { + width: 16px; + height: 16px; +} + +@media (max-width: 920px) { + .directory__filters { + grid-template-columns: 1fr 1fr; + } + + .progress-panel__header, + .directory__header, + .employee-card__header { + align-items: stretch; + flex-direction: column; + } +} + +@media (max-width: 620px) { + .directory__filters { + grid-template-columns: 1fr; + } +} diff --git a/app/templates/dashboard.html b/app/templates/dashboard.html index 3cc1342..b53d35c 100644 --- a/app/templates/dashboard.html +++ b/app/templates/dashboard.html @@ -1,63 +1,69 @@ -{% extends "base.html" %} -{% block title %}Обзор · MIEM Employees{% endblock %} -{% block content %} -
- Всего в базе{{ counts.total }} - Работают{{ counts.active }} - Новые за запуск{{ counts.new_in_last_run }} - Уволены{{ counts.dismissed }} -
-
-
- Последний добавленный - {% if counts.latest_added %} - {{ counts.latest_added.full_name or counts.latest_added.canonical_url }} - {% else %} - Сотрудников пока нет - {% endif %} -
- - Запуски - {{ counts.runs }} - -
- Ошибки - {{ counts.errors }} -
-
-
-
+{% extends "base.html" %} +{% block title %}Обзор · MIEM Employees{% endblock %} +{% block content %} +
+ Всего в базе{{ counts.total }} + Работают{{ counts.active }} + Требуют проверки{{ counts.verification_required }} + Новые за запуск{{ counts.new_in_last_run }} + Уволены{{ counts.dismissed }} +
+
+
+ Последний добавленный + {% if counts.latest_added %} + {{ counts.latest_added.full_name or counts.latest_added.canonical_url }} + {% else %} + Сотрудников пока нет + {% endif %} +
+ + Запуски + {{ counts.runs }} + +
+ Ошибки + {{ counts.errors }} +
+
+
+

Прогресс парсинга

-
- -
-
- {% set run = counts.current_running_run or latest_run %} -
-
- {{ run.status_display if run else "Ожидание" }} - обработано: {{ run.processed_count if run else 0 }} / {{ run.found_count if run else 0 }} - без изменений: {{ run.skipped_count if run else 0 }} - ошибок: {{ run.error_count if run else 0 }} +
+
+ +
+
+ +
-
-
-
-
{{ run.progress_percent if run else 0 }}%
-
-
-
-

Последние запуски

- - - - {% for run in runs %} - - {% endfor %} - -
IDСтатусОбработаноБез измененийОшибкиСтарт
{{ run.id }}{{ run.status_display }}{{ run.parsed_count }}{{ run.skipped_count }}{{ run.error_count }}{{ run.started_display }}
-
-{% endblock %} -{% block scripts %} - -{% endblock %} +
+ {% set run = counts.current_running_run or latest_run %} +
+
+ {{ run.status_display if run else "Ожидание" }} + обработано: {{ run.processed_count if run else 0 }} / {{ run.found_count if run else 0 }} + без изменений: {{ run.skipped_count if run else 0 }} + ошибок: {{ run.error_count if run else 0 }} +
+
+
+
+
{{ run.progress_percent if run else 0 }}%
+
+
+
+

Последние запуски

+ + + + {% for run in runs %} + + {% endfor %} + +
IDСтатусОбработаноБез измененийОшибкиСтарт
{{ run.id }}{{ run.status_display }}{{ run.parsed_count }}{{ run.skipped_count }}{{ run.error_count }}{{ run.started_display }}
+
+{% endblock %} +{% block scripts %} + +{% endblock %} diff --git a/app/version.py b/app/version.py index 8937d1a..dca7a84 100644 --- a/app/version.py +++ b/app/version.py @@ -1,3 +1,3 @@ -APP_VERSION = "0.7.0" -FRONTEND_VERSION = "0.7.0" -BACKEND_VERSION = "0.7.0" +APP_VERSION = "0.7.1" +FRONTEND_VERSION = "0.7.1" +BACKEND_VERSION = "0.7.1" diff --git a/pyproject.toml b/pyproject.toml index a6e5830..7d37163 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,28 +1,28 @@ -[project] -name = "miem-workers" -version = "0.7.0" -description = "MIEM employees parser, admin API, and MCP server" -requires-python = ">=3.11" -dependencies = [ - "apscheduler>=3.10.4", - "beautifulsoup4>=4.12.3", - "fastapi>=0.115.0", - "httpx>=0.27.0", - "jinja2>=3.1.4", - "lxml>=5.2.0", - "psycopg[binary]>=3.2.0", - "pydantic-settings>=2.4.0", - "python-multipart>=0.0.9", - "requests>=2.32.0", - "sqlalchemy>=2.0.32", - "uvicorn[standard]>=0.30.0", -] - -[project.optional-dependencies] -dev = [ - "pytest>=8.3.0", -] - -[tool.pytest.ini_options] -testpaths = ["tests"] -pythonpath = ["."] +[project] +name = "miem-workers" +version = "0.7.1" +description = "MIEM employees parser, admin API, and MCP server" +requires-python = ">=3.11" +dependencies = [ + "apscheduler>=3.10.4", + "beautifulsoup4>=4.12.3", + "fastapi>=0.115.0", + "httpx>=0.27.0", + "jinja2>=3.1.4", + "lxml>=5.2.0", + "psycopg[binary]>=3.2.0", + "pydantic-settings>=2.4.0", + "python-multipart>=0.0.9", + "requests>=2.32.0", + "sqlalchemy>=2.0.32", + "uvicorn[standard]>=0.30.0", +] + +[project.optional-dependencies] +dev = [ + "pytest>=8.3.0", +] + +[tool.pytest.ini_options] +testpaths = ["tests"] +pythonpath = ["."] diff --git a/tests/test_admin_templates.py b/tests/test_admin_templates.py index d0b0679..eef4b49 100644 --- a/tests/test_admin_templates.py +++ b/tests/test_admin_templates.py @@ -1,95 +1,105 @@ -from pathlib import Path - - -def test_base_navigation_is_russian_and_has_no_legacy_employees_link(): - template = Path("app/templates/base.html").read_text(encoding="utf-8") - - assert "Обзор" in template - assert "Сотрудники" in template - assert "Запуски" in template - assert "Выйти" in template - assert 'MIEM Employees' in template - assert ">Employees<" not in template - assert "/admin/employees" not in template - - -def test_directory_template_is_russian_and_uses_display_dates(): - template = Path("app/templates/directory.html").read_text(encoding="utf-8") - - assert "Сотрудники" in template - assert "Колонки" in template - assert "Применить" in template - assert "На странице: {{ value }}" in template - assert "{% for value in [25, 50, 100] %}" in template - assert "Найдено:" in template - assert "Новости" in template - assert "employee.news_count" in template - assert "employee.first_seen_display" in template - assert "employee.last_seen_display" in template - assert "employee.dismissed_display" in template - assert "Directory" not in template - assert "employees found" not in template - - -def test_admin_employees_route_redirects_to_directory(): - source = Path("app/admin.py").read_text(encoding="utf-8") - - assert 'RedirectResponse("/admin/directory", status_code=303)' in source - - -def test_dashboard_limits_latest_runs_to_five(): - source = Path("app/admin.py").read_text(encoding="utf-8") - - assert "order_by(desc(CrawlRun.started_at)).limit(5)" in source - assert "order_by(desc(CrawlRun.started_at)).limit(10)" not in source - - -def test_runs_template_links_to_run_detail(): - template = Path("app/templates/runs.html").read_text(encoding="utf-8") - - assert 'onclick="window.location.href=\'/admin/runs/{{ run.id }}\'"' in template - assert "onkeydown=\"if (event.key === 'Enter' || event.key === ' ')" in template - assert 'role="link"' in template - assert 'tabindex="0"' in template - assert 'data-row-href="/admin/runs/{{ run.id }}"' not in template - assert '' not in template - - -def test_run_detail_template_extends_base_and_shows_change_groups(): - template = Path("app/templates/run_detail.html").read_text(encoding="utf-8") - - assert '{% extends "base.html" %}' in template - assert 'id="new-employees"' in template - assert "Новые сотрудники" in template - assert "Потеряшки" in template - assert "Уволенные" in template - assert "Детализация сотрудников для этого запуска недоступна" in template - - +from pathlib import Path + + +def test_base_navigation_is_russian_and_has_no_legacy_employees_link(): + template = Path("app/templates/base.html").read_text(encoding="utf-8") + + assert "Обзор" in template + assert "Сотрудники" in template + assert "Запуски" in template + assert "Выйти" in template + assert 'MIEM Employees' in template + assert ">Employees<" not in template + assert "/admin/employees" not in template + + +def test_directory_template_is_russian_and_uses_display_dates(): + template = Path("app/templates/directory.html").read_text(encoding="utf-8") + + assert "Сотрудники" in template + assert "Колонки" in template + assert "Применить" in template + assert "На странице: {{ value }}" in template + assert "{% for value in [25, 50, 100] %}" in template + assert "Найдено:" in template + assert "Новости" in template + assert "employee.news_count" in template + assert "employee.first_seen_display" in template + assert "employee.last_seen_display" in template + assert "employee.dismissed_display" in template + assert "verification_required" in template + assert "Directory" not in template + assert "employees found" not in template + + +def test_admin_employees_route_redirects_to_directory(): + source = Path("app/admin.py").read_text(encoding="utf-8") + + assert 'RedirectResponse("/admin/directory", status_code=303)' in source + + +def test_dashboard_limits_latest_runs_to_five(): + source = Path("app/admin.py").read_text(encoding="utf-8") + + assert "order_by(desc(CrawlRun.started_at)).limit(5)" in source + assert "order_by(desc(CrawlRun.started_at)).limit(10)" not in source + + +def test_runs_template_links_to_run_detail(): + template = Path("app/templates/runs.html").read_text(encoding="utf-8") + + assert 'onclick="window.location.href=\'/admin/runs/{{ run.id }}\'"' in template + assert "onkeydown=\"if (event.key === 'Enter' || event.key === ' ')" in template + assert 'role="link"' in template + assert 'tabindex="0"' in template + assert 'data-row-href="/admin/runs/{{ run.id }}"' not in template + assert '' not in template + + +def test_run_detail_template_extends_base_and_shows_change_groups(): + template = Path("app/templates/run_detail.html").read_text(encoding="utf-8") + + assert '{% extends "base.html" %}' in template + assert 'id="new-employees"' in template + assert "Новые сотрудники" in template + assert "Потеряшки" in template + assert "Требуют проверки" in template + assert "Уволенные" in template + assert "Детализация сотрудников для этого запуска недоступна" in template + + def test_dashboard_metric_cards_link_to_admin_targets(): - template = Path("app/templates/dashboard.html").read_text(encoding="utf-8") - - assert 'href="/admin/directory"' in template - assert 'href="/admin/directory?status=active"' in template - assert '/admin/runs/{{ latest_run.id }}#new-employees' in template - assert 'href="/admin/directory?status=dismissed"' in template + template = Path("app/templates/dashboard.html").read_text(encoding="utf-8") + + assert 'href="/admin/directory"' in template + assert 'href="/admin/directory?status=active"' in template + assert 'href="/admin/directory?status=verification_required"' in template + assert '/admin/runs/{{ latest_run.id }}#new-employees' in template + assert 'href="/admin/directory?status=dismissed"' in template assert 'href="/admin/runs"' in template -def test_dashboard_latest_run_rows_link_to_run_detail(): +def test_dashboard_has_dismissed_status_refresh_action(): template = Path("app/templates/dashboard.html").read_text(encoding="utf-8") - assert 'onclick="window.location.href=\'/admin/runs/{{ run.id }}\'"' in template - assert "onkeydown=\"if (event.key === 'Enter' || event.key === ' ')" in template - assert 'role="link"' in template - assert 'tabindex="0"' in template - assert 'data-row-href="/admin/runs/{{ run.id }}"' not in template - assert '' not in template - - -def test_admin_js_supports_keyboard_activation_for_clickable_rows(): - source = Path("app/static/admin.js").read_text(encoding="utf-8") - - assert 'addEventListener("keydown"' in source - assert '"Enter"' in source - assert '" "' in source + assert 'action="/admin/dismissed/refresh"' in template + assert "Проверить уволенных" in template + + +def test_dashboard_latest_run_rows_link_to_run_detail(): + template = Path("app/templates/dashboard.html").read_text(encoding="utf-8") + + assert 'onclick="window.location.href=\'/admin/runs/{{ run.id }}\'"' in template + assert "onkeydown=\"if (event.key === 'Enter' || event.key === ' ')" in template + assert 'role="link"' in template + assert 'tabindex="0"' in template + assert 'data-row-href="/admin/runs/{{ run.id }}"' not in template + assert '' not in template + + +def test_admin_js_supports_keyboard_activation_for_clickable_rows(): + source = Path("app/static/admin.js").read_text(encoding="utf-8") + + assert 'addEventListener("keydown"' in source + assert '"Enter"' in source + assert '" "' in source diff --git a/tests/test_api_mcp.py b/tests/test_api_mcp.py index d2e25f9..55426da 100644 --- a/tests/test_api_mcp.py +++ b/tests/test_api_mcp.py @@ -1,532 +1,532 @@ -import json -from datetime import datetime, timezone -from types import SimpleNamespace - -from fastapi.testclient import TestClient -from sqlalchemy import create_engine, select -from sqlalchemy.orm import sessionmaker -from sqlalchemy.pool import StaticPool - -from app.config import Settings, get_settings -from app.db import Base, get_db -from app.main import app -from app.models import CrawlRun, CrawlRunEmployeeChange, Employee, EmployeePublication -from app.security import SESSION_COOKIE, sign_session - - -def test_health_returns_versions(): - client = TestClient(app) - - response = client.get("/api/health") - - assert response.status_code == 200 - assert response.json()["backend_version"] == "0.7.0" - - -def test_mcp_lists_tools_without_auth_and_ignores_auth_header(): - engine = create_engine( - "sqlite:///:memory:", - connect_args={"check_same_thread": False}, - poolclass=StaticPool, - ) - Base.metadata.create_all(engine) - Session = sessionmaker(bind=engine) - - def override_db(): - session = Session() - try: - yield session - finally: - session.close() - - app.dependency_overrides[get_db] = override_db - client = TestClient(app) - - without_auth = client.post("/mcp", json={"jsonrpc": "2.0", "id": 1, "method": "tools/list", "params": {}}) - with_auth = client.post( - "/mcp", - headers={"Authorization": "Bearer anything"}, - json={"jsonrpc": "2.0", "id": 1, "method": "tools/list", "params": {}}, - ) - - assert without_auth.status_code == 200 - assert with_auth.status_code == 200 - tool_names = {tool["name"] for tool in without_auth.json()["result"]["tools"]} - assert "search_employees" in tool_names - assert "get_service_info" in tool_names - assert "sync_employees" in tool_names - assert any(tool["name"] == "get_crawl_run_details" for tool in without_auth.json()["result"]["tools"]) - assert with_auth.json()["result"]["tools"] == without_auth.json()["result"]["tools"] - - app.dependency_overrides.clear() - - -def test_mcp_search_employees_returns_matching_employee(): - engine = create_engine( - "sqlite:///:memory:", - connect_args={"check_same_thread": False}, - poolclass=StaticPool, - ) - Base.metadata.create_all(engine) - Session = sessionmaker(bind=engine) - session = Session() - session.add( - Employee( - profile_key="staff:avsergeev", - profile_type="staff", - profile_id="avsergeev", - canonical_url="https://www.hse.ru/staff/avsergeev", - full_name="Сергеев Алексей Викторович", - status="active", - first_seen_at=datetime.now(timezone.utc), - last_seen_at=datetime.now(timezone.utc), - current_data={"sections": []}, - ) - ) - session.commit() - session.close() - - def override_db(): - db = Session() - try: - yield db - finally: - db.close() - - app.dependency_overrides[get_db] = override_db - client = TestClient(app) - - response = client.post( - "/mcp", - json={ - "jsonrpc": "2.0", - "id": 1, - "method": "tools/call", - "params": {"name": "search_employees", "arguments": {"query": "Сергеев"}}, - }, - ) - - assert response.status_code == 200 - assert "Сергеев Алексей Викторович" in response.json()["result"]["content"][0]["text"] - - app.dependency_overrides.clear() - - -def test_mcp_service_info_returns_tools_and_dataset_hash(): - engine = create_engine( - "sqlite:///:memory:", - connect_args={"check_same_thread": False}, - poolclass=StaticPool, - ) - Base.metadata.create_all(engine) - Session = sessionmaker(bind=engine) - session = Session() - session.add( - Employee( - profile_key="staff:alpha", - profile_type="staff", - profile_id="alpha", - canonical_url="https://www.hse.ru/staff/alpha", - full_name="Alpha Person", - status="active", - current_checksum="a" * 64, - current_data={"sections": []}, - ) - ) - session.commit() - session.close() - - def override_db(): - db = Session() - try: - yield db - finally: - db.close() - - app.dependency_overrides[get_db] = override_db - client = TestClient(app) - - response = client.post( - "/mcp", - json={"jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": {"name": "get_service_info", "arguments": {}}}, - ) - - assert response.status_code == 200 - payload = json.loads(response.json()["result"]["content"][0]["text"]) - assert payload["service_name"] == "miem-employees" - assert payload["backend_version"] == "0.7.0" - assert payload["dataset"]["hash"] - assert any(tool["name"] == "sync_employees" for tool in payload["tools"]) - - app.dependency_overrides.clear() - - -def test_mcp_list_employee_publications_prefers_stored_publications_with_fallback(): - engine = create_engine( - "sqlite:///:memory:", - connect_args={"check_same_thread": False}, - poolclass=StaticPool, - ) - Base.metadata.create_all(engine) - Session = sessionmaker(bind=engine) - session = Session() - stored_employee = Employee( - profile_key="staff:stored", - profile_type="staff", - profile_id="stored", - canonical_url="https://www.hse.ru/staff/stored", - full_name="Stored Person", - status="active", - current_data={ - "sections": [ - { - "type": "publications", - "publications": [{"title": "Old JSON Publication", "url": "https://example.test/old"}], - } - ] - }, - ) - fallback_employee = Employee( - profile_key="staff:fallback", - profile_type="staff", - profile_id="fallback", - canonical_url="https://www.hse.ru/staff/fallback", - full_name="Fallback Person", - status="active", - current_data={ - "sections": [ - { - "type": "publications", - "publications": [{"title": "Fallback Publication", "url": "https://example.test/fallback"}], - } - ] - }, - ) - session.add_all([stored_employee, fallback_employee]) - session.commit() - session.add( - EmployeePublication( - employee_id=stored_employee.id, - publication_id="pub-1", - title="Stored Publication", - year=2024, - publication_type="ARTICLE", - url="https://publications.hse.ru/view/pub-1", - doi_url="https://doi.org/10.1/test", - citation_text="Stored Citation", - annotation={"ru": "Аннотация", "en": "Abstract"}, - description={"main": "Stored Citation"}, - authors=[{"id": "1", "title_ru": "Автор", "is_current_employee": True}], - source_hash="a" * 64, - ) - ) - session.commit() - session.close() - - def override_db(): - db = Session() - try: - yield db - finally: - db.close() - - app.dependency_overrides[get_db] = override_db - client = TestClient(app) - - stored_response = client.post( - "/mcp", - json={ - "jsonrpc": "2.0", - "id": 1, - "method": "tools/call", - "params": {"name": "list_employee_publications", "arguments": {"profile_id_or_url": "stored"}}, - }, - ) - fallback_response = client.post( - "/mcp", - json={ - "jsonrpc": "2.0", - "id": 2, - "method": "tools/call", - "params": {"name": "list_employee_publications", "arguments": {"profile_id_or_url": "fallback"}}, - }, - ) - - stored_payload = json.loads(stored_response.json()["result"]["content"][0]["text"]) - fallback_payload = json.loads(fallback_response.json()["result"]["content"][0]["text"]) - assert stored_payload["items"][0]["title"] == "Stored Publication" - assert stored_payload["items"][0]["doi_url"] == "https://doi.org/10.1/test" - assert stored_payload["items"][0]["annotation"] == {"ru": "Аннотация", "en": "Abstract"} - assert stored_payload["items"][0]["authors"] == [{"id": "1", "title_ru": "Автор", "is_current_employee": True}] - assert fallback_payload["items"][0]["title"] == "Fallback Publication" - - app.dependency_overrides.clear() - - -def test_mcp_sync_employees_full_empty_and_unknown_hash_modes(): - engine = create_engine( - "sqlite:///:memory:", - connect_args={"check_same_thread": False}, - poolclass=StaticPool, - ) - Base.metadata.create_all(engine) - Session = sessionmaker(bind=engine) - session = Session() - session.add( - Employee( - profile_key="staff:alpha", - profile_type="staff", - profile_id="alpha", - canonical_url="https://www.hse.ru/staff/alpha", - full_name="Alpha Person", - status="active", - current_checksum="a" * 64, - current_data={"sections": [{"type": "paragraphs"}]}, - ) - ) - session.commit() - session.close() - - def override_db(): - db = Session() - try: - yield db - finally: - db.close() - - app.dependency_overrides[get_db] = override_db - client = TestClient(app) - - full_response = client.post( - "/mcp", - json={"jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": {"name": "sync_employees", "arguments": {}}}, - ) - full_payload = json.loads(full_response.json()["result"]["content"][0]["text"]) - current_hash = full_payload["to_hash"] - - empty_response = client.post( - "/mcp", - json={ - "jsonrpc": "2.0", - "id": 2, - "method": "tools/call", - "params": {"name": "sync_employees", "arguments": {"client_hash": current_hash}}, - }, - ) - empty_payload = json.loads(empty_response.json()["result"]["content"][0]["text"]) - - unknown_response = client.post( - "/mcp", - json={ - "jsonrpc": "2.0", - "id": 3, - "method": "tools/call", - "params": {"name": "sync_employees", "arguments": {"client_hash": "missing"}}, - }, - ) - unknown_payload = json.loads(unknown_response.json()["result"]["content"][0]["text"]) - - assert full_payload["mode"] == "full" - assert full_payload["items"][0]["data"] == {"sections": [{"type": "paragraphs"}]} - assert empty_payload["mode"] == "delta" - assert empty_payload["changes"] == {"added": [], "updated": [], "dismissed": [], "removed": []} - assert unknown_payload["mode"] == "full" - assert unknown_payload["reason"] == "unknown_client_hash" - - app.dependency_overrides.clear() - - -def test_mcp_get_crawl_run_details_returns_changes(): - engine = create_engine( - "sqlite:///:memory:", - connect_args={"check_same_thread": False}, - poolclass=StaticPool, - ) - Base.metadata.create_all(engine) - Session = sessionmaker(bind=engine) - session = Session() - run = CrawlRun(source_url="https://miem.hse.ru/persons", status="completed", new_count=1) - employee = Employee( - profile_key="staff:new", - profile_type="staff", - profile_id="new", - canonical_url="https://www.hse.ru/staff/new", - full_name="New Person", - status="active", - first_seen_at=datetime.now(timezone.utc), - last_seen_at=datetime.now(timezone.utc), - ) - session.add_all([run, employee]) - session.commit() - session.add( - CrawlRunEmployeeChange( - crawl_run_id=run.id, - employee_id=employee.id, - profile_key=employee.profile_key, - profile_url=employee.canonical_url, - full_name=employee.full_name, - change_type="new", - profile_available=True, - message="added", - ) - ) - session.commit() - run_id = run.id - session.close() - - def override_db(): - db = Session() - try: - yield db - finally: - db.close() - - app.dependency_overrides[get_db] = override_db - client = TestClient(app) - - response = client.post( - "/mcp", - json={ - "jsonrpc": "2.0", - "id": 1, - "method": "tools/call", - "params": {"name": "get_crawl_run_details", "arguments": {"run_id": run_id}}, - }, - ) - - assert response.status_code == 200 - text = response.json()["result"]["content"][0]["text"] - assert "New Person" in text - assert "changes_detail_available" in text - - app.dependency_overrides.clear() - - -def test_mcp_protected_resource_metadata_route_is_removed(): - client = TestClient(app) - - response = client.get("/.well-known/oauth-protected-resource") - - assert response.status_code == 404 - - -def test_api_employees_and_stats_require_admin_session(): - engine = create_engine( - "sqlite:///:memory:", - connect_args={"check_same_thread": False}, - poolclass=StaticPool, - ) - Base.metadata.create_all(engine) - Session = sessionmaker(bind=engine) - db = Session() - db.add( - Employee( - profile_key="staff:alpha", - profile_type="staff", - profile_id="alpha", - canonical_url="https://www.hse.ru/staff/alpha", - full_name="Alpha Person", - status="active", - first_seen_at=datetime.now(timezone.utc), - last_seen_at=datetime.now(timezone.utc), - current_data={"contacts": {"emails": ["alpha@hse.ru"]}, "sections": []}, - ) - ) - run = CrawlRun(source_url="https://miem.hse.ru/persons", status="completed", new_count=1) - db.add(run) - db.commit() - db.add( - CrawlRunEmployeeChange( - crawl_run_id=run.id, - employee_id=1, - profile_key="staff:alpha", - profile_url="https://www.hse.ru/staff/alpha", - full_name="Alpha Person", - change_type="new", - profile_available=True, - message="added", - ) - ) - db.commit() - run_id = run.id - db.close() - - settings = Settings(admin_username="admin", admin_password="password", session_secret="session-secret") - - def override_db(): - session = Session() - try: - yield session - finally: - session.close() - - app.dependency_overrides[get_db] = override_db - app.dependency_overrides[get_settings] = lambda: settings - client = TestClient(app) - client.cookies.set(SESSION_COOKIE, sign_session("admin", settings)) - - employees = client.get("/api/employees", params={"q": "Alpha", "has_email": True}) - stats = client.get("/api/stats") - run_details = client.get(f"/api/crawl-runs/{run_id}") - - assert employees.status_code == 200 - assert employees.json()["total"] == 1 - assert stats.status_code == 200 - assert stats.json()["new_in_last_run"] == 1 - assert run_details.status_code == 200 - assert run_details.json()["changes"]["new"][0]["full_name"] == "Alpha Person" - - app.dependency_overrides.clear() - - -def test_admin_refresh_employee_route_updates_only_requested_employee(monkeypatch): - engine = create_engine( - "sqlite:///:memory:", - connect_args={"check_same_thread": False}, - poolclass=StaticPool, - ) - Base.metadata.create_all(engine) - Session = sessionmaker(bind=engine) - db = Session() - db.add( - Employee( - profile_key="org_person:133709486", - profile_type="org_person", - profile_id="133709486", - canonical_url="https://www.hse.ru/org/persons/133709486", - full_name="Будков Юрий Алексеевич", - status="active", - ) - ) - db.commit() - employee_id = db.scalar(select(Employee.id)) - db.close() - - settings = Settings(admin_username="admin", admin_password="password", session_secret="session-secret") - - def override_db(): - session = Session() - try: - yield session - finally: - session.close() - - calls = [] - - def fake_refresh_employee(db, refreshed_employee, route_settings): - calls.append((refreshed_employee.id, route_settings)) - return SimpleNamespace(status="completed") - - app.dependency_overrides[get_db] = override_db - app.dependency_overrides[get_settings] = lambda: settings - monkeypatch.setattr("app.admin.refresh_employee", fake_refresh_employee) - client = TestClient(app) - client.cookies.set(SESSION_COOKIE, sign_session("admin", settings)) - - response = client.post(f"/admin/employees/{employee_id}/refresh", follow_redirects=False) - - assert response.status_code == 303 - assert response.headers["location"] == f"/admin/employees/{employee_id}?refresh_status=success" - assert calls == [(employee_id, settings)] - - app.dependency_overrides.clear() +import json +from datetime import datetime, timezone +from types import SimpleNamespace + +from fastapi.testclient import TestClient +from sqlalchemy import create_engine, select +from sqlalchemy.orm import sessionmaker +from sqlalchemy.pool import StaticPool + +from app.config import Settings, get_settings +from app.db import Base, get_db +from app.main import app +from app.models import CrawlRun, CrawlRunEmployeeChange, Employee, EmployeePublication +from app.security import SESSION_COOKIE, sign_session + + +def test_health_returns_versions(): + client = TestClient(app) + + response = client.get("/api/health") + + assert response.status_code == 200 + assert response.json()["backend_version"] == "0.7.1" + + +def test_mcp_lists_tools_without_auth_and_ignores_auth_header(): + engine = create_engine( + "sqlite:///:memory:", + connect_args={"check_same_thread": False}, + poolclass=StaticPool, + ) + Base.metadata.create_all(engine) + Session = sessionmaker(bind=engine) + + def override_db(): + session = Session() + try: + yield session + finally: + session.close() + + app.dependency_overrides[get_db] = override_db + client = TestClient(app) + + without_auth = client.post("/mcp", json={"jsonrpc": "2.0", "id": 1, "method": "tools/list", "params": {}}) + with_auth = client.post( + "/mcp", + headers={"Authorization": "Bearer anything"}, + json={"jsonrpc": "2.0", "id": 1, "method": "tools/list", "params": {}}, + ) + + assert without_auth.status_code == 200 + assert with_auth.status_code == 200 + tool_names = {tool["name"] for tool in without_auth.json()["result"]["tools"]} + assert "search_employees" in tool_names + assert "get_service_info" in tool_names + assert "sync_employees" in tool_names + assert any(tool["name"] == "get_crawl_run_details" for tool in without_auth.json()["result"]["tools"]) + assert with_auth.json()["result"]["tools"] == without_auth.json()["result"]["tools"] + + app.dependency_overrides.clear() + + +def test_mcp_search_employees_returns_matching_employee(): + engine = create_engine( + "sqlite:///:memory:", + connect_args={"check_same_thread": False}, + poolclass=StaticPool, + ) + Base.metadata.create_all(engine) + Session = sessionmaker(bind=engine) + session = Session() + session.add( + Employee( + profile_key="staff:avsergeev", + profile_type="staff", + profile_id="avsergeev", + canonical_url="https://www.hse.ru/staff/avsergeev", + full_name="Сергеев Алексей Викторович", + status="active", + first_seen_at=datetime.now(timezone.utc), + last_seen_at=datetime.now(timezone.utc), + current_data={"sections": []}, + ) + ) + session.commit() + session.close() + + def override_db(): + db = Session() + try: + yield db + finally: + db.close() + + app.dependency_overrides[get_db] = override_db + client = TestClient(app) + + response = client.post( + "/mcp", + json={ + "jsonrpc": "2.0", + "id": 1, + "method": "tools/call", + "params": {"name": "search_employees", "arguments": {"query": "Сергеев"}}, + }, + ) + + assert response.status_code == 200 + assert "Сергеев Алексей Викторович" in response.json()["result"]["content"][0]["text"] + + app.dependency_overrides.clear() + + +def test_mcp_service_info_returns_tools_and_dataset_hash(): + engine = create_engine( + "sqlite:///:memory:", + connect_args={"check_same_thread": False}, + poolclass=StaticPool, + ) + Base.metadata.create_all(engine) + Session = sessionmaker(bind=engine) + session = Session() + session.add( + Employee( + profile_key="staff:alpha", + profile_type="staff", + profile_id="alpha", + canonical_url="https://www.hse.ru/staff/alpha", + full_name="Alpha Person", + status="active", + current_checksum="a" * 64, + current_data={"sections": []}, + ) + ) + session.commit() + session.close() + + def override_db(): + db = Session() + try: + yield db + finally: + db.close() + + app.dependency_overrides[get_db] = override_db + client = TestClient(app) + + response = client.post( + "/mcp", + json={"jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": {"name": "get_service_info", "arguments": {}}}, + ) + + assert response.status_code == 200 + payload = json.loads(response.json()["result"]["content"][0]["text"]) + assert payload["service_name"] == "miem-employees" + assert payload["backend_version"] == "0.7.1" + assert payload["dataset"]["hash"] + assert any(tool["name"] == "sync_employees" for tool in payload["tools"]) + + app.dependency_overrides.clear() + + +def test_mcp_list_employee_publications_prefers_stored_publications_with_fallback(): + engine = create_engine( + "sqlite:///:memory:", + connect_args={"check_same_thread": False}, + poolclass=StaticPool, + ) + Base.metadata.create_all(engine) + Session = sessionmaker(bind=engine) + session = Session() + stored_employee = Employee( + profile_key="staff:stored", + profile_type="staff", + profile_id="stored", + canonical_url="https://www.hse.ru/staff/stored", + full_name="Stored Person", + status="active", + current_data={ + "sections": [ + { + "type": "publications", + "publications": [{"title": "Old JSON Publication", "url": "https://example.test/old"}], + } + ] + }, + ) + fallback_employee = Employee( + profile_key="staff:fallback", + profile_type="staff", + profile_id="fallback", + canonical_url="https://www.hse.ru/staff/fallback", + full_name="Fallback Person", + status="active", + current_data={ + "sections": [ + { + "type": "publications", + "publications": [{"title": "Fallback Publication", "url": "https://example.test/fallback"}], + } + ] + }, + ) + session.add_all([stored_employee, fallback_employee]) + session.commit() + session.add( + EmployeePublication( + employee_id=stored_employee.id, + publication_id="pub-1", + title="Stored Publication", + year=2024, + publication_type="ARTICLE", + url="https://publications.hse.ru/view/pub-1", + doi_url="https://doi.org/10.1/test", + citation_text="Stored Citation", + annotation={"ru": "Аннотация", "en": "Abstract"}, + description={"main": "Stored Citation"}, + authors=[{"id": "1", "title_ru": "Автор", "is_current_employee": True}], + source_hash="a" * 64, + ) + ) + session.commit() + session.close() + + def override_db(): + db = Session() + try: + yield db + finally: + db.close() + + app.dependency_overrides[get_db] = override_db + client = TestClient(app) + + stored_response = client.post( + "/mcp", + json={ + "jsonrpc": "2.0", + "id": 1, + "method": "tools/call", + "params": {"name": "list_employee_publications", "arguments": {"profile_id_or_url": "stored"}}, + }, + ) + fallback_response = client.post( + "/mcp", + json={ + "jsonrpc": "2.0", + "id": 2, + "method": "tools/call", + "params": {"name": "list_employee_publications", "arguments": {"profile_id_or_url": "fallback"}}, + }, + ) + + stored_payload = json.loads(stored_response.json()["result"]["content"][0]["text"]) + fallback_payload = json.loads(fallback_response.json()["result"]["content"][0]["text"]) + assert stored_payload["items"][0]["title"] == "Stored Publication" + assert stored_payload["items"][0]["doi_url"] == "https://doi.org/10.1/test" + assert stored_payload["items"][0]["annotation"] == {"ru": "Аннотация", "en": "Abstract"} + assert stored_payload["items"][0]["authors"] == [{"id": "1", "title_ru": "Автор", "is_current_employee": True}] + assert fallback_payload["items"][0]["title"] == "Fallback Publication" + + app.dependency_overrides.clear() + + +def test_mcp_sync_employees_full_empty_and_unknown_hash_modes(): + engine = create_engine( + "sqlite:///:memory:", + connect_args={"check_same_thread": False}, + poolclass=StaticPool, + ) + Base.metadata.create_all(engine) + Session = sessionmaker(bind=engine) + session = Session() + session.add( + Employee( + profile_key="staff:alpha", + profile_type="staff", + profile_id="alpha", + canonical_url="https://www.hse.ru/staff/alpha", + full_name="Alpha Person", + status="active", + current_checksum="a" * 64, + current_data={"sections": [{"type": "paragraphs"}]}, + ) + ) + session.commit() + session.close() + + def override_db(): + db = Session() + try: + yield db + finally: + db.close() + + app.dependency_overrides[get_db] = override_db + client = TestClient(app) + + full_response = client.post( + "/mcp", + json={"jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": {"name": "sync_employees", "arguments": {}}}, + ) + full_payload = json.loads(full_response.json()["result"]["content"][0]["text"]) + current_hash = full_payload["to_hash"] + + empty_response = client.post( + "/mcp", + json={ + "jsonrpc": "2.0", + "id": 2, + "method": "tools/call", + "params": {"name": "sync_employees", "arguments": {"client_hash": current_hash}}, + }, + ) + empty_payload = json.loads(empty_response.json()["result"]["content"][0]["text"]) + + unknown_response = client.post( + "/mcp", + json={ + "jsonrpc": "2.0", + "id": 3, + "method": "tools/call", + "params": {"name": "sync_employees", "arguments": {"client_hash": "missing"}}, + }, + ) + unknown_payload = json.loads(unknown_response.json()["result"]["content"][0]["text"]) + + assert full_payload["mode"] == "full" + assert full_payload["items"][0]["data"] == {"sections": [{"type": "paragraphs"}]} + assert empty_payload["mode"] == "delta" + assert empty_payload["changes"] == {"added": [], "updated": [], "dismissed": [], "removed": []} + assert unknown_payload["mode"] == "full" + assert unknown_payload["reason"] == "unknown_client_hash" + + app.dependency_overrides.clear() + + +def test_mcp_get_crawl_run_details_returns_changes(): + engine = create_engine( + "sqlite:///:memory:", + connect_args={"check_same_thread": False}, + poolclass=StaticPool, + ) + Base.metadata.create_all(engine) + Session = sessionmaker(bind=engine) + session = Session() + run = CrawlRun(source_url="https://miem.hse.ru/persons", status="completed", new_count=1) + employee = Employee( + profile_key="staff:new", + profile_type="staff", + profile_id="new", + canonical_url="https://www.hse.ru/staff/new", + full_name="New Person", + status="active", + first_seen_at=datetime.now(timezone.utc), + last_seen_at=datetime.now(timezone.utc), + ) + session.add_all([run, employee]) + session.commit() + session.add( + CrawlRunEmployeeChange( + crawl_run_id=run.id, + employee_id=employee.id, + profile_key=employee.profile_key, + profile_url=employee.canonical_url, + full_name=employee.full_name, + change_type="new", + profile_available=True, + message="added", + ) + ) + session.commit() + run_id = run.id + session.close() + + def override_db(): + db = Session() + try: + yield db + finally: + db.close() + + app.dependency_overrides[get_db] = override_db + client = TestClient(app) + + response = client.post( + "/mcp", + json={ + "jsonrpc": "2.0", + "id": 1, + "method": "tools/call", + "params": {"name": "get_crawl_run_details", "arguments": {"run_id": run_id}}, + }, + ) + + assert response.status_code == 200 + text = response.json()["result"]["content"][0]["text"] + assert "New Person" in text + assert "changes_detail_available" in text + + app.dependency_overrides.clear() + + +def test_mcp_protected_resource_metadata_route_is_removed(): + client = TestClient(app) + + response = client.get("/.well-known/oauth-protected-resource") + + assert response.status_code == 404 + + +def test_api_employees_and_stats_require_admin_session(): + engine = create_engine( + "sqlite:///:memory:", + connect_args={"check_same_thread": False}, + poolclass=StaticPool, + ) + Base.metadata.create_all(engine) + Session = sessionmaker(bind=engine) + db = Session() + db.add( + Employee( + profile_key="staff:alpha", + profile_type="staff", + profile_id="alpha", + canonical_url="https://www.hse.ru/staff/alpha", + full_name="Alpha Person", + status="active", + first_seen_at=datetime.now(timezone.utc), + last_seen_at=datetime.now(timezone.utc), + current_data={"contacts": {"emails": ["alpha@hse.ru"]}, "sections": []}, + ) + ) + run = CrawlRun(source_url="https://miem.hse.ru/persons", status="completed", new_count=1) + db.add(run) + db.commit() + db.add( + CrawlRunEmployeeChange( + crawl_run_id=run.id, + employee_id=1, + profile_key="staff:alpha", + profile_url="https://www.hse.ru/staff/alpha", + full_name="Alpha Person", + change_type="new", + profile_available=True, + message="added", + ) + ) + db.commit() + run_id = run.id + db.close() + + settings = Settings(admin_username="admin", admin_password="password", session_secret="session-secret") + + def override_db(): + session = Session() + try: + yield session + finally: + session.close() + + app.dependency_overrides[get_db] = override_db + app.dependency_overrides[get_settings] = lambda: settings + client = TestClient(app) + client.cookies.set(SESSION_COOKIE, sign_session("admin", settings)) + + employees = client.get("/api/employees", params={"q": "Alpha", "has_email": True}) + stats = client.get("/api/stats") + run_details = client.get(f"/api/crawl-runs/{run_id}") + + assert employees.status_code == 200 + assert employees.json()["total"] == 1 + assert stats.status_code == 200 + assert stats.json()["new_in_last_run"] == 1 + assert run_details.status_code == 200 + assert run_details.json()["changes"]["new"][0]["full_name"] == "Alpha Person" + + app.dependency_overrides.clear() + + +def test_admin_refresh_employee_route_updates_only_requested_employee(monkeypatch): + engine = create_engine( + "sqlite:///:memory:", + connect_args={"check_same_thread": False}, + poolclass=StaticPool, + ) + Base.metadata.create_all(engine) + Session = sessionmaker(bind=engine) + db = Session() + db.add( + Employee( + profile_key="org_person:133709486", + profile_type="org_person", + profile_id="133709486", + canonical_url="https://www.hse.ru/org/persons/133709486", + full_name="Будков Юрий Алексеевич", + status="active", + ) + ) + db.commit() + employee_id = db.scalar(select(Employee.id)) + db.close() + + settings = Settings(admin_username="admin", admin_password="password", session_secret="session-secret") + + def override_db(): + session = Session() + try: + yield session + finally: + session.close() + + calls = [] + + def fake_refresh_employee(db, refreshed_employee, route_settings): + calls.append((refreshed_employee.id, route_settings)) + return SimpleNamespace(status="completed") + + app.dependency_overrides[get_db] = override_db + app.dependency_overrides[get_settings] = lambda: settings + monkeypatch.setattr("app.admin.refresh_employee", fake_refresh_employee) + client = TestClient(app) + client.cookies.set(SESSION_COOKIE, sign_session("admin", settings)) + + response = client.post(f"/admin/employees/{employee_id}/refresh", follow_redirects=False) + + assert response.status_code == 303 + assert response.headers["location"] == f"/admin/employees/{employee_id}?refresh_status=success" + assert calls == [(employee_id, settings)] + + app.dependency_overrides.clear() diff --git a/tests/test_crawler.py b/tests/test_crawler.py index d40da22..cd34bea 100644 --- a/tests/test_crawler.py +++ b/tests/test_crawler.py @@ -1,383 +1,569 @@ -import gzip -from datetime import datetime, timezone - +import gzip +from datetime import datetime, timezone + from app.models import ( - CrawlError, - CrawlRun, - CrawlRunEmployeeChange, - Employee, - EmployeeNewsLink, - EmployeePublication, - EmployeeSnapshot, - ParseResourceCache, + CrawlError, + CrawlRun, + CrawlRunEmployeeChange, + Employee, + EmployeeNewsLink, + EmployeePublication, + EmployeeSnapshot, + ParseResourceCache, ) -from app.services.crawler import _checksum, _mark_dismissed, _upsert_employee -from app.services.resource_cache import ResourceCache - - -class FakeResponse: - def __init__(self, status_code): - self.status_code = status_code - - -class FakeSession: - def __init__(self, statuses): - self.statuses = statuses - - def get(self, url, **_kwargs): - return FakeResponse(self.statuses[url]) - - -class ConditionalResponse: - def __init__(self, status_code, text="", headers=None): - self.status_code = status_code - self._text = text - self.headers = headers or {} - self.text_read = False - - @property - def text(self): - self.text_read = True - return self._text - - def raise_for_status(self): - return None - - +from app.config import Settings +from app.services.crawler import _checksum, _mark_dismissed, _upsert_employee, refresh_dismissed_status +from app.services.resource_cache import ResourceCache + + +class FakeResponse: + def __init__(self, status_code): + self.status_code = status_code + + +class FakeSession: + def __init__(self, statuses): + self.statuses = statuses + + def get(self, url, **_kwargs): + return FakeResponse(self.statuses[url]) + + +class ConditionalResponse: + def __init__(self, status_code, text="", headers=None): + self.status_code = status_code + self._text = text + self.headers = headers or {} + self.text_read = False + + @property + def text(self): + self.text_read = True + return self._text + + def raise_for_status(self): + return None + + class ConditionalSession: - def __init__(self): - self.requests = [] - self.not_modified_response = ConditionalResponse(304) - - def get(self, url, **kwargs): - self.requests.append((url, kwargs)) - if kwargs["headers"].get("If-None-Match") == '"cached"': - return self.not_modified_response + def __init__(self): + self.requests = [] + self.not_modified_response = ConditionalResponse(304) + + def get(self, url, **kwargs): + self.requests.append((url, kwargs)) + if kwargs["headers"].get("If-None-Match") == '"cached"': + return self.not_modified_response return ConditionalResponse(200, "fresh", {"ETag": '"fresh"'}) -def test_mark_dismissed_records_missing_source_when_profile_is_available(db_session): - run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") - db_session.add(run) - db_session.add( - Employee( - profile_key="staff:kept", - canonical_url="https://www.hse.ru/staff/kept", - status="active", - first_seen_at=datetime.now(timezone.utc), - last_seen_at=datetime.now(timezone.utc), - ) +def test_refresh_dismissed_status_reactivates_only_profiles_in_source(monkeypatch, db_session): + now = datetime.now(timezone.utc) + found = Employee( + profile_key="staff:returned", + canonical_url="https://www.hse.ru/staff/returned", + status="dismissed", + dismissed_at=now, + first_seen_at=now, + last_seen_at=now, ) - db_session.add( - Employee( - profile_key="staff:missing", - canonical_url="https://www.hse.ru/staff/missing", - status="active", - first_seen_at=datetime.now(timezone.utc), - last_seen_at=datetime.now(timezone.utc), - ) - ) - db_session.commit() - - dismissed = _mark_dismissed( - db_session, - run, - {"staff:kept"}, - FakeSession({"https://www.hse.ru/staff/missing": 200}), - 30, - ) - - assert dismissed == 0 - assert db_session.query(Employee).filter_by(profile_key="staff:kept").one().status == "active" - missing = db_session.query(Employee).filter_by(profile_key="staff:missing").one() - assert missing.status == "active" - assert missing.dismissed_at is None - change = db_session.query(CrawlRunEmployeeChange).one() - assert change.change_type == "missing_from_source" - assert change.profile_available is True - - -def test_mark_dismissed_marks_missing_employee_when_profile_is_unavailable(db_session): - run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") - employee = Employee( + still_dismissed = Employee( profile_key="staff:gone", canonical_url="https://www.hse.ru/staff/gone", - status="active", - first_seen_at=datetime.now(timezone.utc), - last_seen_at=datetime.now(timezone.utc), + status="dismissed", + dismissed_at=now, + first_seen_at=now, + last_seen_at=now, ) - db_session.add_all([run, employee]) + db_session.add_all([found, still_dismissed]) db_session.commit() - - dismissed = _mark_dismissed( - db_session, - run, - set(), - FakeSession({"https://www.hse.ru/staff/gone": 404}), - 30, + monkeypatch.setattr( + "app.services.crawler.collect_profile_links", + lambda *_args, **_kwargs: ["https://www.hse.ru/staff/returned"], ) - assert dismissed == 1 - assert employee.status == "dismissed" - assert employee.dismissed_at is not None - change = db_session.query(CrawlRunEmployeeChange).one() - assert change.change_type == "dismissed" - assert change.profile_available is False + run = refresh_dismissed_status(db_session, Settings()) - -def test_upsert_employee_increments_new_count_and_records_change_for_new_employee(db_session): - run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") - db_session.add(run) - db_session.commit() - - _upsert_employee( - db_session, - run, - { - "source_url": "https://www.hse.ru/staff/newperson", - "profile_type": "staff", - "profile_id": "newperson", - "full_name": "New Person", - "tabs": [], - "sections": [], - "parser_version": "0.2.0", - "_html": "", - }, - ) - db_session.commit() - - assert run.new_count == 1 - change = db_session.query(CrawlRunEmployeeChange).one() - assert change.change_type == "new" - assert change.full_name == "New Person" - - -def test_resource_cache_uses_etag_and_reuses_cached_body_on_304(db_session): - db_session.add( - ParseResourceCache( - profile_key="staff:cached", - resource_key="main-html", - method="GET", - url="https://www.hse.ru/staff/cached", - request_fingerprint="020d59db7b358d9023d0f185bcbf5a9c085d3cf2bf91d92d48eee9147e8d0f01", - etag='"cached"', - body_hash="cached-hash", - body_snapshot=gzip.compress("cached body".encode("utf-8")), - parser_version="0.6.0", - ) - ) - db_session.commit() - session = ConditionalSession() - - result = ResourceCache(db_session).fetch_text( - session, - profile_key="staff:cached", - resource_key="main-html", - method="GET", - url="https://www.hse.ru/staff/cached", - headers={"User-Agent": "test"}, - timeout=10, - ) - - assert session.requests[0][1]["headers"]["If-None-Match"] == '"cached"' - assert result.text == "cached body" - assert result.from_cache is True - assert session.not_modified_response.text_read is False - - -def test_upsert_employee_skips_snapshot_when_checksum_is_unchanged(db_session): - first_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") - second_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") - db_session.add_all([first_run, second_run]) - db_session.commit() - - _, first_changed = _upsert_employee(db_session, first_run, _parsed_employee("same")) - _, second_changed = _upsert_employee(db_session, second_run, _parsed_employee("same")) - db_session.commit() - - assert first_changed is True - assert second_changed is False - assert db_session.query(EmployeeSnapshot).count() == 1 - - -def test_upsert_employee_saves_publications_and_reuses_existing_rows(db_session): - first_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") - second_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") - db_session.add_all([first_run, second_run]) - db_session.commit() - - parsed = _parsed_employee("published") - parsed["sections"] = [ - { - "type": "publications", - "publications": [ - { - "id": "888959076", - "publication_id": "888959076", - "title": "Detailed Publication", - "year": 2023, - "publication_type": "ARTICLE", - "language": "ru", - "status": 1, - "url": "https://publications.hse.ru/view/888959076", - "doi_url": "https://doi.org/10.1/test", - "citation_text": "Detailed citation", - "annotation": {"ru": "Аннотация"}, - "description": {"main": "Detailed citation"}, - "authors": [{"id": "1", "title_ru": "Автор"}], - "raw_data": {"id": "888959076", "title": "Detailed Publication"}, - } - ], - } - ] - - employee, _ = _upsert_employee(db_session, first_run, parsed) - db_session.commit() - _upsert_employee(db_session, second_run, _parsed_employee_with_publication("published")) - db_session.commit() - - publications = db_session.query(EmployeePublication).filter_by(employee_id=employee.id).all() - assert len(publications) == 1 - assert publications[0].doi_url == "https://doi.org/10.1/test" - assert publications[0].authors == [{"id": "1", "title_ru": "Автор"}] - - -def test_upsert_employee_records_publication_errors_without_failing_employee(monkeypatch, db_session): - run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") - db_session.add(run) - db_session.commit() - - def broken_sync(*_args, **_kwargs): - raise RuntimeError("boom") - - monkeypatch.setattr("app.services.crawler._sync_employee_publications", broken_sync) - - employee, changed = _upsert_employee(db_session, run, _parsed_employee_with_publication("error-safe")) - db_session.commit() - - assert changed is True - assert employee.full_name == "Same Person" - assert db_session.query(Employee).filter_by(profile_key="staff:error-safe").one() - error = db_session.query(CrawlError).one() - assert "публикации" in error.message.lower() - - -def test_upsert_employee_saves_news_links_and_reuses_existing_rows(db_session): - first_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") - second_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") - db_session.add_all([first_run, second_run]) - db_session.commit() - - employee, _ = _upsert_employee(db_session, first_run, _parsed_employee_with_news("news-person")) - db_session.commit() - _upsert_employee(db_session, second_run, _parsed_employee_with_news("news-person")) - db_session.commit() - - news_links = db_session.query(EmployeeNewsLink).filter_by(employee_id=employee.id).all() - assert len(news_links) == 1 - assert news_links[0].title == "News Title" - assert news_links[0].url == "https://www.hse.ru/news/1.html" - assert news_links[0].published_year == 2026 - - -def test_upsert_employee_records_news_errors_without_failing_employee(monkeypatch, db_session): - run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") - db_session.add(run) - db_session.commit() - - def broken_sync(*_args, **_kwargs): - raise RuntimeError("boom") - - monkeypatch.setattr("app.services.crawler._sync_employee_news_links", broken_sync) - - employee, changed = _upsert_employee(db_session, run, _parsed_employee_with_news("news-error-safe")) - db_session.commit() - - assert changed is True - assert employee.full_name == "Same Person" - assert db_session.query(Employee).filter_by(profile_key="staff:news-error-safe").one() - error = db_session.query(CrawlError).one() - assert "новости" in error.message.lower() - - -def test_checksum_changes_when_widget_data_changes(): - base = _parsed_employee("widgets") - changed = _parsed_employee("widgets") - changed["sections"] = [ - { - "type": "publications", - "publications": [{"id": "1", "title": "New publication"}], - } - ] - - assert _checksum(base) != _checksum(changed) - - -def test_checksum_ignores_date_dependent_experience_text(): - first = _parsed_employee("experience") - second = _parsed_employee("experience") - first["sections"] = [{"raw_text": "Стаж работы в НИУ ВШЭ: 5 лет"}] - second["sections"] = [{"raw_text": "Стаж работы в НИУ ВШЭ: 6 лет"}] - - assert _checksum(first) == _checksum(second) - - -def _parsed_employee(profile_id: str) -> dict: - return { - "source_url": f"https://www.hse.ru/staff/{profile_id}", - "profile_type": "staff", - "profile_id": profile_id, - "full_name": "Same Person", - "tabs": [], - "sections": [], - "parser_version": "0.6.0", - "_html": "", - } - - -def _parsed_employee_with_publication(profile_id: str) -> dict: - parsed = _parsed_employee(profile_id) - parsed["sections"] = [ - { - "type": "publications", - "publications": [ - { - "id": "888959076", - "publication_id": "888959076", - "title": "Detailed Publication", - "year": 2023, - "publication_type": "ARTICLE", - "language": "ru", - "status": 1, - "url": "https://publications.hse.ru/view/888959076", - "doi_url": "https://doi.org/10.1/test", - "citation_text": "Detailed citation", - "annotation": {"ru": "Аннотация"}, - "description": {"main": "Detailed citation"}, - "authors": [{"id": "1", "title_ru": "Автор"}], - "raw_data": {"id": "888959076", "title": "Detailed Publication"}, - } - ], - } - ] - return parsed - - -def _parsed_employee_with_news(profile_id: str) -> dict: - parsed = _parsed_employee(profile_id) - parsed["sections"] = [ - { - "type": "news", - "news_links": [ - { - "title": "News Title", - "url": "https://www.hse.ru/news/1.html", - "summary": "News summary", - "published_at": "2026-04-28T00:00:00+00:00", - "published_year": 2026, - "raw_data": {"title": "News Title", "url": "https://www.hse.ru/news/1.html"}, - } - ], - } - ] - return parsed + assert run.status == "completed" + assert run.parsed_count == 1 + assert run.skipped_count == 1 + assert found.status == "active" + assert found.dismissed_at is None + assert still_dismissed.status == "dismissed" + + +def test_mark_dismissed_records_missing_source_when_profile_is_available(db_session): + run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") + db_session.add(run) + db_session.add( + Employee( + profile_key="staff:kept", + canonical_url="https://www.hse.ru/staff/kept", + status="active", + first_seen_at=datetime.now(timezone.utc), + last_seen_at=datetime.now(timezone.utc), + ) + ) + db_session.add( + Employee( + profile_key="staff:missing", + canonical_url="https://www.hse.ru/staff/missing", + status="active", + first_seen_at=datetime.now(timezone.utc), + last_seen_at=datetime.now(timezone.utc), + ) + ) + db_session.commit() + + dismissed = _mark_dismissed( + db_session, + run, + {"staff:kept"}, + FakeSession({"https://www.hse.ru/staff/missing": 200}), + 30, + ) + + assert dismissed == 0 + assert db_session.query(Employee).filter_by(profile_key="staff:kept").one().status == "active" + missing = db_session.query(Employee).filter_by(profile_key="staff:missing").one() + assert missing.status == "active" + assert missing.dismissed_at is None + change = db_session.query(CrawlRunEmployeeChange).one() + assert change.change_type == "missing_from_source" + assert change.profile_available is True + + +def test_mark_dismissed_requires_consecutive_unavailable_checks(db_session): + run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") + employee = Employee( + profile_key="staff:gone", + canonical_url="https://www.hse.ru/staff/gone", + status="active", + first_seen_at=datetime.now(timezone.utc), + last_seen_at=datetime.now(timezone.utc), + ) + db_session.add_all([run, employee]) + db_session.commit() + + first_check = _mark_dismissed( + db_session, + run, + set(), + FakeSession({"https://www.hse.ru/staff/gone": 404}), + 30, + confirmation_runs=2, + ) + + assert first_check == 0 + assert employee.status == "verification_required" + assert employee.dismissed_at is None + assert employee.profile_unavailable_streak == 1 + assert db_session.query(CrawlRunEmployeeChange).one().change_type == "verification_required" + + second_check = _mark_dismissed( + db_session, + run, + set(), + FakeSession({"https://www.hse.ru/staff/gone": 404}), + 30, + confirmation_runs=2, + ) + + assert second_check == 1 + assert employee.status == "dismissed" + assert employee.dismissed_at is not None + assert employee.profile_unavailable_streak == 2 + change = db_session.query(CrawlRunEmployeeChange).order_by(CrawlRunEmployeeChange.id).all()[-1] + assert change.change_type == "dismissed" + assert change.profile_available is False + + +def test_mark_dismissed_does_not_dismiss_on_server_error(db_session): + run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") + employee = Employee( + profile_key="staff:temporary-error", + canonical_url="https://www.hse.ru/staff/temporary-error", + status="active", + first_seen_at=datetime.now(timezone.utc), + last_seen_at=datetime.now(timezone.utc), + ) + db_session.add_all([run, employee]) + db_session.commit() + + dismissed = _mark_dismissed( + db_session, + run, + set(), + FakeSession({"https://www.hse.ru/staff/temporary-error": 503}), + 30, + confirmation_runs=1, + ) + + assert dismissed == 0 + assert employee.status == "active" + assert employee.profile_unavailable_streak == 0 + assert db_session.query(CrawlError).one().error_type == "ProfileAvailabilityCheckError" + + +def test_available_profile_resets_verification_streak(db_session): + run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") + employee = Employee( + profile_key="staff:restored", + canonical_url="https://www.hse.ru/staff/restored", + status="active", + first_seen_at=datetime.now(timezone.utc), + last_seen_at=datetime.now(timezone.utc), + ) + db_session.add_all([run, employee]) + db_session.commit() + + _mark_dismissed( + db_session, + run, + set(), + FakeSession({"https://www.hse.ru/staff/restored": 404}), + 30, + confirmation_runs=3, + ) + _mark_dismissed( + db_session, + run, + set(), + FakeSession({"https://www.hse.ru/staff/restored": 200}), + 30, + confirmation_runs=3, + ) + + assert employee.status == "active" + assert employee.profile_unavailable_streak == 0 + + +def test_mark_dismissed_blocks_mass_auto_dismissals(db_session): + run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") + employees = [ + Employee( + profile_key=f"staff:gone-{index}", + canonical_url=f"https://www.hse.ru/staff/gone-{index}", + status="active", + first_seen_at=datetime.now(timezone.utc), + last_seen_at=datetime.now(timezone.utc), + ) + for index in range(2) + ] + db_session.add_all([run, *employees]) + db_session.commit() + + dismissed = _mark_dismissed( + db_session, + run, + set(), + FakeSession({employee.canonical_url: 404 for employee in employees}), + 30, + confirmation_runs=1, + max_auto_dismissals=1, + ) + + assert dismissed == 0 + assert {employee.status for employee in employees} == {"verification_required"} + assert "приостановлено" in run.message + + +def test_upsert_employee_increments_new_count_and_records_change_for_new_employee(db_session): + run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") + db_session.add(run) + db_session.commit() + + _upsert_employee( + db_session, + run, + { + "source_url": "https://www.hse.ru/staff/newperson", + "profile_type": "staff", + "profile_id": "newperson", + "full_name": "New Person", + "tabs": [], + "sections": [], + "parser_version": "0.2.0", + "_html": "", + }, + ) + db_session.commit() + + assert run.new_count == 1 + change = db_session.query(CrawlRunEmployeeChange).one() + assert change.change_type == "new" + assert change.full_name == "New Person" + + +def test_upsert_employee_reconciles_profile_moved_to_new_url(db_session): + run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") + employee = Employee( + profile_key="staff:abelov", + canonical_url="https://www.hse.ru/staff/abelov", + full_name="Белов Александр Владимирович", + status="active", + first_seen_at=datetime.now(timezone.utc), + last_seen_at=datetime.now(timezone.utc), + ) + db_session.add_all([run, employee]) + db_session.commit() + employee_id = employee.id + + updated, changed = _upsert_employee( + db_session, + run, + { + "source_url": "https://www.hse.ru/org/persons/47634735", + "profile_type": "org_person", + "profile_id": "47634735", + "full_name": "Белов Александр Владимирович", + "tabs": [], + "sections": [], + "parser_version": "0.7.0", + "_html": "", + }, + ) + db_session.commit() + + assert changed is True + assert updated.id == employee_id + assert updated.profile_key == "org_person:47634735" + assert updated.canonical_url == "https://www.hse.ru/org/persons/47634735" + assert updated.status == "active" + assert run.new_count == 0 + assert db_session.query(Employee).count() == 1 + assert {item.url for item in updated.profile_urls} == { + "https://www.hse.ru/staff/abelov", + "https://www.hse.ru/org/persons/47634735", + } + + +def test_resource_cache_uses_etag_and_reuses_cached_body_on_304(db_session): + db_session.add( + ParseResourceCache( + profile_key="staff:cached", + resource_key="main-html", + method="GET", + url="https://www.hse.ru/staff/cached", + request_fingerprint="020d59db7b358d9023d0f185bcbf5a9c085d3cf2bf91d92d48eee9147e8d0f01", + etag='"cached"', + body_hash="cached-hash", + body_snapshot=gzip.compress("cached body".encode("utf-8")), + parser_version="0.6.0", + ) + ) + db_session.commit() + session = ConditionalSession() + + result = ResourceCache(db_session).fetch_text( + session, + profile_key="staff:cached", + resource_key="main-html", + method="GET", + url="https://www.hse.ru/staff/cached", + headers={"User-Agent": "test"}, + timeout=10, + ) + + assert session.requests[0][1]["headers"]["If-None-Match"] == '"cached"' + assert result.text == "cached body" + assert result.from_cache is True + assert session.not_modified_response.text_read is False + + +def test_upsert_employee_skips_snapshot_when_checksum_is_unchanged(db_session): + first_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") + second_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") + db_session.add_all([first_run, second_run]) + db_session.commit() + + _, first_changed = _upsert_employee(db_session, first_run, _parsed_employee("same")) + _, second_changed = _upsert_employee(db_session, second_run, _parsed_employee("same")) + db_session.commit() + + assert first_changed is True + assert second_changed is False + assert db_session.query(EmployeeSnapshot).count() == 1 + + +def test_upsert_employee_saves_publications_and_reuses_existing_rows(db_session): + first_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") + second_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") + db_session.add_all([first_run, second_run]) + db_session.commit() + + parsed = _parsed_employee("published") + parsed["sections"] = [ + { + "type": "publications", + "publications": [ + { + "id": "888959076", + "publication_id": "888959076", + "title": "Detailed Publication", + "year": 2023, + "publication_type": "ARTICLE", + "language": "ru", + "status": 1, + "url": "https://publications.hse.ru/view/888959076", + "doi_url": "https://doi.org/10.1/test", + "citation_text": "Detailed citation", + "annotation": {"ru": "Аннотация"}, + "description": {"main": "Detailed citation"}, + "authors": [{"id": "1", "title_ru": "Автор"}], + "raw_data": {"id": "888959076", "title": "Detailed Publication"}, + } + ], + } + ] + + employee, _ = _upsert_employee(db_session, first_run, parsed) + db_session.commit() + _upsert_employee(db_session, second_run, _parsed_employee_with_publication("published")) + db_session.commit() + + publications = db_session.query(EmployeePublication).filter_by(employee_id=employee.id).all() + assert len(publications) == 1 + assert publications[0].doi_url == "https://doi.org/10.1/test" + assert publications[0].authors == [{"id": "1", "title_ru": "Автор"}] + + +def test_upsert_employee_records_publication_errors_without_failing_employee(monkeypatch, db_session): + run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") + db_session.add(run) + db_session.commit() + + def broken_sync(*_args, **_kwargs): + raise RuntimeError("boom") + + monkeypatch.setattr("app.services.crawler._sync_employee_publications", broken_sync) + + employee, changed = _upsert_employee(db_session, run, _parsed_employee_with_publication("error-safe")) + db_session.commit() + + assert changed is True + assert employee.full_name == "Same Person" + assert db_session.query(Employee).filter_by(profile_key="staff:error-safe").one() + error = db_session.query(CrawlError).one() + assert "публикации" in error.message.lower() + + +def test_upsert_employee_saves_news_links_and_reuses_existing_rows(db_session): + first_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") + second_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") + db_session.add_all([first_run, second_run]) + db_session.commit() + + employee, _ = _upsert_employee(db_session, first_run, _parsed_employee_with_news("news-person")) + db_session.commit() + _upsert_employee(db_session, second_run, _parsed_employee_with_news("news-person")) + db_session.commit() + + news_links = db_session.query(EmployeeNewsLink).filter_by(employee_id=employee.id).all() + assert len(news_links) == 1 + assert news_links[0].title == "News Title" + assert news_links[0].url == "https://www.hse.ru/news/1.html" + assert news_links[0].published_year == 2026 + + +def test_upsert_employee_records_news_errors_without_failing_employee(monkeypatch, db_session): + run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") + db_session.add(run) + db_session.commit() + + def broken_sync(*_args, **_kwargs): + raise RuntimeError("boom") + + monkeypatch.setattr("app.services.crawler._sync_employee_news_links", broken_sync) + + employee, changed = _upsert_employee(db_session, run, _parsed_employee_with_news("news-error-safe")) + db_session.commit() + + assert changed is True + assert employee.full_name == "Same Person" + assert db_session.query(Employee).filter_by(profile_key="staff:news-error-safe").one() + error = db_session.query(CrawlError).one() + assert "новости" in error.message.lower() + + +def test_checksum_changes_when_widget_data_changes(): + base = _parsed_employee("widgets") + changed = _parsed_employee("widgets") + changed["sections"] = [ + { + "type": "publications", + "publications": [{"id": "1", "title": "New publication"}], + } + ] + + assert _checksum(base) != _checksum(changed) + + +def test_checksum_ignores_date_dependent_experience_text(): + first = _parsed_employee("experience") + second = _parsed_employee("experience") + first["sections"] = [{"raw_text": "Стаж работы в НИУ ВШЭ: 5 лет"}] + second["sections"] = [{"raw_text": "Стаж работы в НИУ ВШЭ: 6 лет"}] + + assert _checksum(first) == _checksum(second) + + +def _parsed_employee(profile_id: str) -> dict: + return { + "source_url": f"https://www.hse.ru/staff/{profile_id}", + "profile_type": "staff", + "profile_id": profile_id, + "full_name": "Same Person", + "tabs": [], + "sections": [], + "parser_version": "0.6.0", + "_html": "", + } + + +def _parsed_employee_with_publication(profile_id: str) -> dict: + parsed = _parsed_employee(profile_id) + parsed["sections"] = [ + { + "type": "publications", + "publications": [ + { + "id": "888959076", + "publication_id": "888959076", + "title": "Detailed Publication", + "year": 2023, + "publication_type": "ARTICLE", + "language": "ru", + "status": 1, + "url": "https://publications.hse.ru/view/888959076", + "doi_url": "https://doi.org/10.1/test", + "citation_text": "Detailed citation", + "annotation": {"ru": "Аннотация"}, + "description": {"main": "Detailed citation"}, + "authors": [{"id": "1", "title_ru": "Автор"}], + "raw_data": {"id": "888959076", "title": "Detailed Publication"}, + } + ], + } + ] + return parsed + + +def _parsed_employee_with_news(profile_id: str) -> dict: + parsed = _parsed_employee(profile_id) + parsed["sections"] = [ + { + "type": "news", + "news_links": [ + { + "title": "News Title", + "url": "https://www.hse.ru/news/1.html", + "summary": "News summary", + "published_at": "2026-04-28T00:00:00+00:00", + "published_year": 2026, + "raw_data": {"title": "News Title", "url": "https://www.hse.ru/news/1.html"}, + } + ], + } + ] + return parsed