admin/miem_workers
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add dismissed employee status refresh #29

Merged
admin merged 1 commits from feature/check-dismissed-employees into main 2026-08-17 16:11:41 +00:00
Conversation 0 Commits 1 Files Changed 11 +3062 -2602
11 changed files with 3062 additions and 2602 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
CHANGELOG.md Normal file
View File

@@ -0,0 +1,5 @@
# Changelog
## 0.7.1
- Добавлена кнопка «Проверить уволенных» для принудительной сверки статуса уволенных сотрудников с текущим списком источника.

290
README.md
View File

@@ -1,142 +1,150 @@
# MIEM Employees Server # MIEM Employees Server
Сервис собирает сотрудников МИЭМ с сайта ВШЭ, хранит карточки и историю обновлений в Postgres, показывает минимальную админку и отдает read-only MCP endpoint для ИИ-агентов. Сервис собирает сотрудников МИЭМ с сайта ВШЭ, хранит карточки и историю обновлений в Postgres, показывает минимальную админку и отдает read-only MCP endpoint для ИИ-агентов.
## Архитектура ## Архитектура
- `api`: FastAPI, REST API, HTML-админка, healthcheck. - `api`: FastAPI, REST API, HTML-админка, healthcheck.
- `worker`: weekly scheduler, который запускает парсинг по `CRAWL_CRON`. - `worker`: weekly scheduler, который запускает парсинг по `CRAWL_CRON`.
- `mcp`: открытый HTTP MCP endpoint для ИИ-агентов. - `mcp`: открытый HTTP MCP endpoint для ИИ-агентов.
- `postgres`: основная БД. - `postgres`: основная БД.
Парсер использует фиксированный источник сотрудников, по умолчанию `https://miem.hse.ru/persons`. Для каждой карточки сохраняются ФИО, должности, год начала работы, контакты, идентификаторы, вкладки профиля, секции, публикации, курсы, ВКР, новости, JSON-снапшот и сжатый HTML-снапшот. Детальные публикации дополнительно нормализуются в отдельную таблицу `employee_publications`, а новости из блока «В новостях» — в `employee_news_links`. Ссылки обходятся только из меню профиля самого сотрудника (`person-menu`), например `#sci`, `#teaching`, `#main`. Парсер использует фиксированный источник сотрудников, по умолчанию `https://miem.hse.ru/persons`. Для каждой карточки сохраняются ФИО, должности, год начала работы, контакты, идентификаторы, вкладки профиля, секции, публикации, курсы, ВКР, новости, JSON-снапшот и сжатый HTML-снапшот. Детальные публикации дополнительно нормализуются в отдельную таблицу `employee_publications`, а новости из блока «В новостях» — в `employee_news_links`. Ссылки обходятся только из меню профиля самого сотрудника (`person-menu`), например `#sci`, `#teaching`, `#main`.
## Переменные окружения ## Переменные окружения
Скопируйте `.env.example` в `.env` и поменяйте секреты: Скопируйте `.env.example` в `.env` и поменяйте секреты:
```bash ```bash
cp .env.example .env cp .env.example .env
``` ```
Основные настройки: Основные настройки:
- `DATABASE_URL`: строка подключения SQLAlchemy. - `DATABASE_URL`: строка подключения SQLAlchemy.
- `SOURCE_URL`: список сотрудников МИЭМ. - `SOURCE_URL`: список сотрудников МИЭМ.
- `CRAWL_CRON`: расписание в формате crontab, по умолчанию `0 3 * * 1`. - `CRAWL_CRON`: расписание в формате crontab, по умолчанию `0 3 * * 1`.
- `CRAWL_LIMIT`: опциональный лимит профилей для тестового запуска. - `CRAWL_LIMIT`: опциональный лимит профилей для тестового запуска.
- `ADMIN_USERNAME`, `ADMIN_PASSWORD`: логин и пароль админки. - `ADMIN_USERNAME`, `ADMIN_PASSWORD`: логин и пароль админки.
- `SESSION_SECRET`: секрет подписи cookie. - `SESSION_SECRET`: секрет подписи cookie.
- `PARSER_USE_PLAYWRIGHT`: включение Playwright-рендера динамических вкладок. - `PARSER_USE_PLAYWRIGHT`: включение Playwright-рендера динамических вкладок.
- `DISMISSAL_CONFIRMATION_RUNS`: сколько последовательных проверок недоступности нужно для увольнения, по умолчанию `3`.
## Локальный запуск - `MAX_AUTO_DISMISSALS_PER_RUN`: защитный лимит массовых автоматических увольнений за один запуск, по умолчанию `25`.
```bash ## Локальный запуск
python -m venv .venv
.venv\Scripts\activate ```bash
pip install -r requirements.txt python -m venv .venv
uvicorn app.main:app --reload .venv\Scripts\activate
``` pip install -r requirements.txt
uvicorn app.main:app --reload
Админка: `http://localhost:8000/admin`. ```
В админке доступны: Админка: `http://localhost:8000/admin`.
- `Dashboard`: общая статистика, последний добавленный сотрудник, прогресс текущего/последнего парсинга и ручной запуск. В админке доступны:
- `Directory`: настраиваемая таблица сотрудников с фильтрами, сортировкой, пагинацией и выбором колонок.
- `Runs`: история запусков, ошибки и progress bar. - `Dashboard`: общая статистика, последний добавленный сотрудник, прогресс текущего/последнего парсинга и ручной запуск.
- `Directory`: настраиваемая таблица сотрудников с фильтрами, сортировкой, пагинацией и выбором колонок.
## Docker Compose - `Runs`: история запусков, ошибки и progress bar.
```bash ## Docker Compose
docker compose up --build
``` ```bash
docker compose up --build
По умолчанию: ```
- API и админка: `http://localhost:8000` По умолчанию:
- MCP: `http://localhost:8001/mcp`
- Postgres: `localhost:5432` - API и админка: `http://localhost:8000`
- MCP: `http://localhost:8001/mcp`
Таблицы создаются приложением при старте. При обновлении существующей базы приложение также добавляет недостающие runtime-колонки, например `crawl_runs.skipped_count`. SQL-миграции для ручного применения лежат в `migrations/`. - Postgres: `localhost:5432`
## Наполнение БД Таблицы создаются приложением при старте. При обновлении существующей базы приложение также добавляет недостающие runtime-колонки, например `crawl_runs.skipped_count`. SQL-миграции для ручного применения лежат в `migrations/`.
Основная карточка сотрудника хранится в `employees`: профиль, статус, даты обнаружения/увольнения, текущий JSON `current_data`, checksum и версия парсера. История успешных изменений сохраняется в `employee_snapshots` вместе с JSON-снимком и сжатым HTML профиля. ## Наполнение БД
Публикации теперь хранятся в двух видах: Основная карточка сотрудника хранится в `employees`: профиль, статус, даты обнаружения/увольнения, текущий JSON `current_data`, checksum и версия парсера. История успешных изменений сохраняется в `employee_snapshots` вместе с JSON-снимком и сжатым HTML профиля.
- краткий список остается внутри `employees.current_data.sections[].publications` для обратной совместимости; Публикации теперь хранятся в двух видах:
- детальные записи сохраняются в `employee_publications` и связываются с сотрудником через `employee_id`.
- краткий список остается внутри `employees.current_data.sections[].publications` для обратной совместимости;
`employee_publications` содержит `publication_id`, название, год, тип публикации, язык, статус, ссылку на карточку HSE Publications, DOI, внешние/document-ссылки, citation text, аннотацию, описание, авторов, raw JSON ответа `searchPubs` и `source_hash` для безопасного повторного upsert. Уникальность поддерживается по `(employee_id, publication_id)` и `(employee_id, source_hash)`, поэтому повторный crawl не должен создавать дубликаты. - детальные записи сохраняются в `employee_publications` и связываются с сотрудником через `employee_id`.
`list_employee_publications` сначала читает `employee_publications`; если детальных строк еще нет, возвращает старые публикации из `current_data`. `employee_publications` содержит `publication_id`, название, год, тип публикации, язык, статус, ссылку на карточку HSE Publications, DOI, внешние/document-ссылки, citation text, аннотацию, описание, авторов, raw JSON ответа `searchPubs` и `source_hash` для безопасного повторного upsert. Уникальность поддерживается по `(employee_id, publication_id)` и `(employee_id, source_hash)`, поэтому повторный crawl не должен создавать дубликаты.
Новости сотрудников также хранятся в двух видах: `list_employee_publications` сначала читает `employee_publications`; если детальных строк еще нет, возвращает старые публикации из `current_data`.
- краткий список остается внутри `employees.current_data.sections[].news_links`; Новости сотрудников также хранятся в двух видах:
- нормализованные карточки из вкладки «В новостях» сохраняются в `employee_news_links`.
- краткий список остается внутри `employees.current_data.sections[].news_links`;
`employee_news_links` содержит название новости, ссылку, краткое описание, дату публикации, год публикации, raw JSON карточки и `source_hash`. Уникальность поддерживается по `(employee_id, url)` и `(employee_id, source_hash)`, поэтому повторный crawl не создает дубликаты. - нормализованные карточки из вкладки «В новостях» сохраняются в `employee_news_links`.
## Парсинг `employee_news_links` содержит название новости, ссылку, краткое описание, дату публикации, год публикации, raw JSON карточки и `source_hash`. Уникальность поддерживается по `(employee_id, url)` и `(employee_id, source_hash)`, поэтому повторный crawl не создает дубликаты.
Weekly worker запускается по `CRAWL_CRON`. Ручной запуск доступен в админке на `Dashboard` и странице `Runs` или через REST: ## Парсинг
```bash Weekly worker запускается по `CRAWL_CRON`. Ручной запуск доступен в админке на `Dashboard` и странице `Runs` или через REST:
curl -X POST http://localhost:8000/api/crawl-runs --cookie "miem_admin_session=..."
``` ```bash
curl -X POST http://localhost:8000/api/crawl-runs --cookie "miem_admin_session=..."
Алгоритм обновления: ```
- найденные сотрудники получают статус `active` и обновленный `last_seen_at`; Алгоритм обновления:
- новые сотрудники добавляются в `employees`;
- количество новых сотрудников за запуск сохраняется в `crawl_runs.new_count`; - найденные сотрудники получают статус `active` и обновленный `last_seen_at`;
- публикации из HSE Publications записываются в `employee_publications`, а краткий список остается в JSON профиля; - новые сотрудники добавляются в `employees`;
- новости из блока «В новостях» записываются в `employee_news_links`, а краткий список остается в JSON профиля; - если профиль перенесен на другой URL, он сопоставляется с прежней записью по единственному точному совпадению ФИО;
- активные сотрудники, исчезнувшие из текущего списка источника, получают статус `dismissed` и `dismissed_at`; - старые URL сохраняются в истории `employee_profile_urls`;
- количество новых сотрудников за запуск сохраняется в `crawl_runs.new_count`;
- публикации из HSE Publications записываются в `employee_publications`, а краткий список остается в JSON профиля;
- новости из блока «В новостях» записываются в `employee_news_links`, а краткий список остается в JSON профиля;
- один `404` старого профиля переводит сотрудника в статус `verification_required`, а не в `dismissed`;
- статус `dismissed` устанавливается только после нескольких последовательных проверок `404`/`410`;
- сетевые ошибки и ответы `5xx` не считаются подтверждением увольнения;
- если число кандидатов на увольнение превышает защитный лимит, автоматическое увольнение приостанавливается;
- кнопка «Проверить уволенных» сверяет только их profile_key с текущим списком источника и возвращает найденных сотрудников в `active` без обновления содержимого профиля;
- каждый успешный новый или измененный разбор сохраняет запись в `employee_snapshots`; - каждый успешный новый или измененный разбор сохраняет запись в `employee_snapshots`;
- неизмененные профили учитываются в `crawl_runs.skipped_count` и не получают новый snapshot. - неизмененные профили учитываются в `crawl_runs.skipped_count` и не получают новый snapshot.
Во время выполнения парсинга `found_count`, `parsed_count`, `skipped_count` и `error_count` обновляются в базе. Админка опрашивает `/api/crawl-runs/latest` и показывает прогресс как `(parsed_count + skipped_count + error_count) / found_count`. Во время выполнения парсинга `found_count`, `parsed_count`, `skipped_count` и `error_count` обновляются в базе. Админка опрашивает `/api/crawl-runs/latest` и показывает прогресс как `(parsed_count + skipped_count + error_count) / found_count`.
## MCP ## MCP
Endpoint: `POST /mcp`, без авторизации на уровне приложения. Endpoint: `POST /mcp`, без авторизации на уровне приложения.
Поддерживаемые tools: Поддерживаемые tools:
- `get_service_info()` - `get_service_info()`
- `sync_employees(client_hash?, include_data?)` - `sync_employees(client_hash?, include_data?)`
- `search_employees(query, status?, limit?)` - `search_employees(query, status?, limit?)`
- `get_employee(profile_id_or_url)` - `get_employee(profile_id_or_url)`
- `list_employee_publications(profile_id_or_url)` — публикации сотрудника; при наличии данных из `employee_publications` возвращает авторов, DOI, аннотацию, описание, citation text, год, тип, язык, статус и ссылку HSE Publications. - `list_employee_publications(profile_id_or_url)` — публикации сотрудника; при наличии данных из `employee_publications` возвращает авторов, DOI, аннотацию, описание, citation text, год, тип, язык, статус и ссылку HSE Publications.
- `list_employee_courses(profile_id_or_url)` - `list_employee_courses(profile_id_or_url)`
- `get_crawl_status()` - `get_crawl_status()`
- `get_crawl_run_details(run_id)` - `get_crawl_run_details(run_id)`
`get_service_info` возвращает метаданные сервиса, список tools и текущую версию набора сотрудников. `sync_employees` отдает полный snapshot или delta по `client_hash`; checksum набора строится по сотрудникам, их статусам и текущим checksums. Ответы tools возвращаются как JSON-строка внутри MCP `content[0].text`. `get_service_info` возвращает метаданные сервиса, список tools и текущую версию набора сотрудников. `sync_employees` отдает полный snapshot или delta по `client_hash`; checksum набора строится по сотрудникам, их статусам и текущим checksums. Ответы tools возвращаются как JSON-строка внутри MCP `content[0].text`.
Новости сотрудника отдельной MCP tool не имеют: они доступны в `get_employee(...).data.sections` и `sync_employees(include_data=true)` как секция `type = "news"` с массивом `news_links`. Новости сотрудника отдельной MCP tool не имеют: они доступны в `get_employee(...).data.sections` и `sync_employees(include_data=true)` как секция `type = "news"` с массивом `news_links`.
Пример локального запроса списка tools: Пример локального запроса списка tools:
```bash ```bash
curl http://localhost:8001/mcp \ curl http://localhost:8001/mcp \
-H "Content-Type: application/json" \ -H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}' -d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}'
``` ```
Если MCP нужно ограничить, делайте это на сетевом уровне: localhost binding, VPN, firewall, reverse proxy или другой внешний контур доступа. Если MCP нужно ограничить, делайте это на сетевом уровне: localhost binding, VPN, firewall, reverse proxy или другой внешний контур доступа.
## Обслуживание ## Обслуживание
```bash ```bash
docker compose logs -f api docker compose logs -f api
docker compose logs -f worker docker compose logs -f worker
docker compose exec postgres pg_dump -U miem miem_workers > backup.sql docker compose exec postgres pg_dump -U miem miem_workers > backup.sql
docker compose down docker compose down
``` ```
Версия сервиса: `0.7.0`. Админка всегда показывает версии backend и frontend в footer. Версия сервиса: `0.7.1`. Админка всегда показывает версии backend и frontend в footer.

489
app/admin.py
View File

@@ -1,219 +1,238 @@
from fastapi import APIRouter, BackgroundTasks, Depends, Form, Request from fastapi import APIRouter, BackgroundTasks, Depends, Form, Request
from fastapi.responses import HTMLResponse, RedirectResponse from fastapi.responses import HTMLResponse, RedirectResponse
from fastapi.templating import Jinja2Templates from fastapi.templating import Jinja2Templates
from sqlalchemy import desc, func, select from sqlalchemy import desc, func, select
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
from app.config import Settings, get_settings from app.config import Settings, get_settings
from app.db import SessionLocal, get_db from app.db import SessionLocal, get_db
from app.models import CrawlError, CrawlRun, Employee from app.models import CrawlError, CrawlRun, Employee
from app.security import SESSION_COOKIE, require_admin, sign_session, verify_admin from app.security import SESSION_COOKIE, require_admin, sign_session, verify_admin
from app.services.admin_data import ( from app.services.admin_data import (
employee_detail_payload, employee_detail_payload,
format_admin_datetime, format_admin_datetime,
list_employees_page, list_employees_page,
run_detail_payload, run_detail_payload,
run_payload, run_payload,
stats_payload, stats_payload,
) )
from app.services.crawl_control import get_running_run, run_crawl_if_idle from app.services.crawl_control import get_running_run, run_crawl_if_idle
from app.services.crawler import refresh_employee from app.services.crawler import refresh_dismissed_status, refresh_employee
from app.version import BACKEND_VERSION, FRONTEND_VERSION from app.version import BACKEND_VERSION, FRONTEND_VERSION
router = APIRouter(prefix="/admin") router = APIRouter(prefix="/admin")
templates = Jinja2Templates(directory="app/templates") templates = Jinja2Templates(directory="app/templates")
@router.get("", response_class=HTMLResponse) @router.get("", response_class=HTMLResponse)
def dashboard(request: Request, db: Session = Depends(get_db), settings: Settings = Depends(get_settings)): def dashboard(request: Request, db: Session = Depends(get_db), settings: Settings = Depends(get_settings)):
require_admin(request, settings) require_admin(request, settings)
counts = stats_payload(db) counts = stats_payload(db)
counts["runs"] = db.scalar(select(func.count()).select_from(CrawlRun)) or 0 counts["runs"] = db.scalar(select(func.count()).select_from(CrawlRun)) or 0
counts["errors"] = db.scalar(select(func.count()).select_from(CrawlError)) or 0 counts["errors"] = db.scalar(select(func.count()).select_from(CrawlError)) or 0
run_models = db.scalars(select(CrawlRun).order_by(desc(CrawlRun.started_at)).limit(5)).all() run_models = db.scalars(select(CrawlRun).order_by(desc(CrawlRun.started_at)).limit(5)).all()
runs = [run_payload(run) for run in run_models] runs = [run_payload(run) for run in run_models]
return _render(request, "dashboard.html", {"counts": counts, "runs": runs, "latest_run": runs[0] if runs else None}) return _render(request, "dashboard.html", {"counts": counts, "runs": runs, "latest_run": runs[0] if runs else None})
@router.get("/login", response_class=HTMLResponse) @router.get("/login", response_class=HTMLResponse)
def login_form(request: Request): def login_form(request: Request):
return _render(request, "login.html", {"error": None}) return _render(request, "login.html", {"error": None})
@router.post("/login") @router.post("/login")
def login( def login(
request: Request, request: Request,
username: str = Form(...), username: str = Form(...),
password: str = Form(...), password: str = Form(...),
settings: Settings = Depends(get_settings), settings: Settings = Depends(get_settings),
): ):
if not verify_admin(username, password, settings): if not verify_admin(username, password, settings):
return _render(request, "login.html", {"error": "Неверный логин или пароль"}, status_code=401) return _render(request, "login.html", {"error": "Неверный логин или пароль"}, status_code=401)
redirect = RedirectResponse("/admin", status_code=303) redirect = RedirectResponse("/admin", status_code=303)
redirect.set_cookie(SESSION_COOKIE, sign_session(username, settings), httponly=True, samesite="lax") redirect.set_cookie(SESSION_COOKIE, sign_session(username, settings), httponly=True, samesite="lax")
return redirect return redirect
@router.post("/logout") @router.post("/logout")
def logout(): def logout():
redirect = RedirectResponse("/admin/login", status_code=303) redirect = RedirectResponse("/admin/login", status_code=303)
redirect.delete_cookie(SESSION_COOKIE) redirect.delete_cookie(SESSION_COOKIE)
return redirect return redirect
@router.get("/employees", response_class=HTMLResponse) @router.get("/employees", response_class=HTMLResponse)
def employees( def employees(
request: Request, request: Request,
status: str | None = None, status: str | None = None,
q: str | None = None, q: str | None = None,
settings: Settings = Depends(get_settings), settings: Settings = Depends(get_settings),
): ):
require_admin(request, settings) require_admin(request, settings)
return RedirectResponse("/admin/directory", status_code=303) return RedirectResponse("/admin/directory", status_code=303)
@router.get("/directory", response_class=HTMLResponse) @router.get("/directory", response_class=HTMLResponse)
def directory( def directory(
request: Request, request: Request,
status: str | None = None, status: str | None = None,
q: str | None = None, q: str | None = None,
started_from: str | None = None, started_from: str | None = None,
started_to: str | None = None, started_to: str | None = None,
has_email: str | None = None, has_email: str | None = None,
sort: str = "full_name", sort: str = "full_name",
direction: str = "asc", direction: str = "asc",
limit: int = 50, limit: int = 50,
offset: int = 0, offset: int = 0,
db: Session = Depends(get_db), db: Session = Depends(get_db),
settings: Settings = Depends(get_settings), settings: Settings = Depends(get_settings),
): ):
require_admin(request, settings) require_admin(request, settings)
parsed_started_from = _parse_date(started_from) parsed_started_from = _parse_date(started_from)
parsed_started_to = _parse_date(started_to) parsed_started_to = _parse_date(started_to)
parsed_has_email = None if has_email in (None, "") else has_email == "true" parsed_has_email = None if has_email in (None, "") else has_email == "true"
page = list_employees_page( page = list_employees_page(
db, db,
status=status, status=status,
q=q, q=q,
started_from=parsed_started_from, started_from=parsed_started_from,
started_to=parsed_started_to, started_to=parsed_started_to,
has_email=parsed_has_email, has_email=parsed_has_email,
sort=sort, sort=sort,
direction=direction, direction=direction,
limit=limit, limit=limit,
offset=offset, offset=offset,
) )
return _render( return _render(
request, request,
"directory.html", "directory.html",
{ {
"page": page, "page": page,
"filters": { "filters": {
"status": status or "", "status": status or "",
"q": q or "", "q": q or "",
"started_from": started_from or "", "started_from": started_from or "",
"started_to": started_to or "", "started_to": started_to or "",
"has_email": has_email or "", "has_email": has_email or "",
"sort": sort, "sort": sort,
"direction": direction, "direction": direction,
"limit": page["limit"], "limit": page["limit"],
"offset": offset, "offset": offset,
}, },
}, },
) )
@router.get("/employees/{employee_id}", response_class=HTMLResponse) @router.get("/employees/{employee_id}", response_class=HTMLResponse)
def employee_detail( def employee_detail(
employee_id: int, employee_id: int,
request: Request, request: Request,
db: Session = Depends(get_db), db: Session = Depends(get_db),
settings: Settings = Depends(get_settings), settings: Settings = Depends(get_settings),
): ):
require_admin(request, settings) require_admin(request, settings)
employee = db.get(Employee, employee_id) employee = db.get(Employee, employee_id)
if not employee: if not employee:
return RedirectResponse("/admin/employees", status_code=303) return RedirectResponse("/admin/employees", status_code=303)
snapshots = [ snapshots = [
{ {
"captured_display": format_admin_datetime(snapshot.captured_at), "captured_display": format_admin_datetime(snapshot.captured_at),
"checksum": snapshot.checksum, "checksum": snapshot.checksum,
"parser_version": snapshot.parser_version, "parser_version": snapshot.parser_version,
} }
for snapshot in sorted(employee.snapshots, key=lambda item: item.captured_at, reverse=True)[:20] for snapshot in sorted(employee.snapshots, key=lambda item: item.captured_at, reverse=True)[:20]
] ]
return _render( return _render(
request, request,
"employee_detail.html", "employee_detail.html",
{ {
"employee": employee, "employee": employee,
"employee_view": employee_detail_payload(employee), "employee_view": employee_detail_payload(employee),
"snapshots": snapshots, "snapshots": snapshots,
"refresh_status": request.query_params.get("refresh_status"), "refresh_status": request.query_params.get("refresh_status"),
}, },
) )
@router.post("/employees/{employee_id}/refresh") @router.post("/employees/{employee_id}/refresh")
def refresh_employee_detail( def refresh_employee_detail(
employee_id: int, employee_id: int,
request: Request, request: Request,
db: Session = Depends(get_db), db: Session = Depends(get_db),
settings: Settings = Depends(get_settings), settings: Settings = Depends(get_settings),
): ):
require_admin(request, settings) require_admin(request, settings)
employee = db.get(Employee, employee_id) employee = db.get(Employee, employee_id)
if not employee: if not employee:
return RedirectResponse("/admin/directory", status_code=303) return RedirectResponse("/admin/directory", status_code=303)
run = refresh_employee(db, employee, settings) run = refresh_employee(db, employee, settings)
status = "success" if run.status == "completed" else "error" status = "success" if run.status == "completed" else "error"
return RedirectResponse(f"/admin/employees/{employee_id}?refresh_status={status}", status_code=303) return RedirectResponse(f"/admin/employees/{employee_id}?refresh_status={status}", status_code=303)
@router.get("/runs", response_class=HTMLResponse) @router.get("/runs", response_class=HTMLResponse)
def runs(request: Request, db: Session = Depends(get_db), settings: Settings = Depends(get_settings)): def runs(request: Request, db: Session = Depends(get_db), settings: Settings = Depends(get_settings)):
require_admin(request, settings) require_admin(request, settings)
run_models = db.scalars(select(CrawlRun).order_by(desc(CrawlRun.started_at)).limit(50)).all() run_models = db.scalars(select(CrawlRun).order_by(desc(CrawlRun.started_at)).limit(50)).all()
items = [run_payload(run) for run in run_models] items = [run_payload(run) for run in run_models]
errors = db.scalars(select(CrawlError).order_by(desc(CrawlError.created_at)).limit(50)).all() errors = db.scalars(select(CrawlError).order_by(desc(CrawlError.created_at)).limit(50)).all()
return _render(request, "runs.html", {"runs": items, "errors": errors}) return _render(request, "runs.html", {"runs": items, "errors": errors})
@router.get("/runs/{run_id}", response_class=HTMLResponse) @router.get("/runs/{run_id}", response_class=HTMLResponse)
def run_detail( def run_detail(
run_id: int, run_id: int,
request: Request, request: Request,
db: Session = Depends(get_db), db: Session = Depends(get_db),
settings: Settings = Depends(get_settings), settings: Settings = Depends(get_settings),
): ):
require_admin(request, settings) require_admin(request, settings)
run = db.get(CrawlRun, run_id) run = db.get(CrawlRun, run_id)
if not run: if not run:
return RedirectResponse("/admin/runs", status_code=303) return RedirectResponse("/admin/runs", status_code=303)
return _render(request, "run_detail.html", {"run": run_detail_payload(db, run)}) return _render(request, "run_detail.html", {"run": run_detail_payload(db, run)})
@router.post("/runs") @router.post("/runs")
def trigger_run( def trigger_run(
request: Request, request: Request,
background_tasks: BackgroundTasks, background_tasks: BackgroundTasks,
db: Session = Depends(get_db), db: Session = Depends(get_db),
settings: Settings = Depends(get_settings), settings: Settings = Depends(get_settings),
): ):
require_admin(request, settings) require_admin(request, settings)
if get_running_run(db): if get_running_run(db):
return RedirectResponse("/admin/runs", status_code=303) return RedirectResponse("/admin/runs", status_code=303)
def _crawl() -> None: def _crawl() -> None:
with SessionLocal() as db: with SessionLocal() as db:
run_crawl_if_idle(db, settings) run_crawl_if_idle(db, settings)
background_tasks.add_task(_crawl) background_tasks.add_task(_crawl)
return RedirectResponse("/admin/runs", status_code=303) return RedirectResponse("/admin/runs", status_code=303)
@router.post("/crawl-now") @router.post("/crawl-now")
def crawl_now( def crawl_now(
request: Request,
background_tasks: BackgroundTasks,
db: Session = Depends(get_db),
settings: Settings = Depends(get_settings),
):
require_admin(request, settings)
if get_running_run(db):
return RedirectResponse("/admin", status_code=303)
def _crawl() -> None:
with SessionLocal() as db:
run_crawl_if_idle(db, settings)
background_tasks.add_task(_crawl)
return RedirectResponse("/admin", status_code=303)
@router.post("/dismissed/refresh")
def refresh_dismissed(
request: Request, request: Request,
background_tasks: BackgroundTasks, background_tasks: BackgroundTasks,
db: Session = Depends(get_db), db: Session = Depends(get_db),
@@ -223,30 +242,30 @@ def crawl_now(
if get_running_run(db): if get_running_run(db):
return RedirectResponse("/admin", status_code=303) return RedirectResponse("/admin", status_code=303)
def _crawl() -> None: def _refresh() -> None:
with SessionLocal() as db: with SessionLocal() as db:
run_crawl_if_idle(db, settings) refresh_dismissed_status(db, settings)
background_tasks.add_task(_crawl) background_tasks.add_task(_refresh)
return RedirectResponse("/admin", status_code=303) return RedirectResponse("/admin", status_code=303)
def _render(request: Request, template: str, context: dict, status_code: int = 200) -> HTMLResponse: def _render(request: Request, template: str, context: dict, status_code: int = 200) -> HTMLResponse:
payload = { payload = {
"request": request, "request": request,
"backend_version": BACKEND_VERSION, "backend_version": BACKEND_VERSION,
"frontend_version": FRONTEND_VERSION, "frontend_version": FRONTEND_VERSION,
**context, **context,
} }
return templates.TemplateResponse(request, template, payload, status_code=status_code) return templates.TemplateResponse(request, template, payload, status_code=status_code)
def _parse_date(value: str | None): def _parse_date(value: str | None):
if not value: if not value:
return None return None
try: try:
from datetime import date from datetime import date
return date.fromisoformat(value) return date.fromisoformat(value)
except ValueError: except ValueError:
return None return None

1228
app/services/crawler.py
View File

File diff suppressed because it is too large Load Diff

1286
app/static/admin.css
View File

File diff suppressed because it is too large Load Diff

128
app/templates/dashboard.html
View File

@@ -1,63 +1,69 @@
{% extends "base.html" %} {% extends "base.html" %}
{% block title %}Обзор · MIEM Employees{% endblock %} {% block title %}Обзор · MIEM Employees{% endblock %}
{% block content %} {% block content %}
<section class="admin__grid"> <section class="admin__grid">
<a class="metric metric--link" href="/admin/directory"><span class="metric__label">Всего в базе</span><span class="metric__value">{{ counts.total }}</span></a> <a class="metric metric--link" href="/admin/directory"><span class="metric__label">Всего в базе</span><span class="metric__value">{{ counts.total }}</span></a>
<a class="metric metric--link" href="/admin/directory?status=active"><span class="metric__label">Работают</span><span class="metric__value">{{ counts.active }}</span></a> <a class="metric metric--link" href="/admin/directory?status=active"><span class="metric__label">Работают</span><span class="metric__value">{{ counts.active }}</span></a>
<a class="metric metric--link" href="{% if latest_run %}/admin/runs/{{ latest_run.id }}#new-employees{% else %}/admin/runs{% endif %}"><span class="metric__label">Новые за запуск</span><span class="metric__value">{{ counts.new_in_last_run }}</span></a> <a class="metric metric--link" href="/admin/directory?status=verification_required"><span class="metric__label">Требуют проверки</span><span class="metric__value">{{ counts.verification_required }}</span></a>
<a class="metric metric--link" href="/admin/directory?status=dismissed"><span class="metric__label">Уволены</span><span class="metric__value">{{ counts.dismissed }}</span></a> <a class="metric metric--link" href="{% if latest_run %}/admin/runs/{{ latest_run.id }}#new-employees{% else %}/admin/runs{% endif %}"><span class="metric__label">Новые за запуск</span><span class="metric__value">{{ counts.new_in_last_run }}</span></a>
</section> <a class="metric metric--link" href="/admin/directory?status=dismissed"><span class="metric__label">Уволены</span><span class="metric__value">{{ counts.dismissed }}</span></a>
<section class="stats-strip"> </section>
<div class="stats-strip__item"> <section class="stats-strip">
<span class="stats-strip__label">Последний добавленный</span> <div class="stats-strip__item">
{% if counts.latest_added %} <span class="stats-strip__label">Последний добавленный</span>
<a class="stats-strip__value" href="/admin/employees/{{ counts.latest_added.id }}">{{ counts.latest_added.full_name or counts.latest_added.canonical_url }}</a> {% if counts.latest_added %}
{% else %} <a class="stats-strip__value" href="/admin/employees/{{ counts.latest_added.id }}">{{ counts.latest_added.full_name or counts.latest_added.canonical_url }}</a>
<span class="stats-strip__value">Сотрудников пока нет</span> {% else %}
{% endif %} <span class="stats-strip__value">Сотрудников пока нет</span>
</div> {% endif %}
<a class="stats-strip__item stats-strip__item--link" href="/admin/runs"> </div>
<span class="stats-strip__label">Запуски</span> <a class="stats-strip__item stats-strip__item--link" href="/admin/runs">
<span class="stats-strip__value">{{ counts.runs }}</span> <span class="stats-strip__label">Запуски</span>
</a> <span class="stats-strip__value">{{ counts.runs }}</span>
<div class="stats-strip__item"> </a>
<span class="stats-strip__label">Ошибки</span> <div class="stats-strip__item">
<span class="stats-strip__value">{{ counts.errors }}</span> <span class="stats-strip__label">Ошибки</span>
</div> <span class="stats-strip__value">{{ counts.errors }}</span>
</section> </div>
<section class="panel progress-panel" data-progress-panel> </section>
<div class="progress-panel__header"> <section class="panel progress-panel" data-progress-panel>
<div class="progress-panel__header">
<h2 class="panel__title">Прогресс парсинга</h2> <h2 class="panel__title">Прогресс парсинга</h2>
<form method="post" action="/admin/crawl-now"> <div class="progress-panel__actions">
<button class="button" type="submit">Запустить парсинг</button> <form method="post" action="/admin/crawl-now">
</form> <button class="button" type="submit">Запустить парсинг</button>
</div> </form>
{% set run = counts.current_running_run or latest_run %} <form method="post" action="/admin/dismissed/refresh">
<div class="progress-panel__body" data-progress-body> <button class="button button--secondary" type="submit">Проверить уволенных</button>
<div class="progress-panel__meta"> </form>
<span data-progress-status>{{ run.status_display if run else "Ожидание" }}</span>
<span>обработано: <span data-progress-processed>{{ run.processed_count if run else 0 }}</span> / <span data-progress-found>{{ run.found_count if run else 0 }}</span></span>
<span>без изменений: <span data-progress-skipped>{{ run.skipped_count if run else 0 }}</span></span>
<span>ошибок: <span data-progress-errors>{{ run.error_count if run else 0 }}</span></span>
</div> </div>
<div class="progress-bar" aria-label="Parsing progress"> </div>
<div class="progress-bar__fill" data-progress-fill style="width: {{ run.progress_percent if run else 0 }}%"></div> {% set run = counts.current_running_run or latest_run %}
</div> <div class="progress-panel__body" data-progress-body>
<div class="progress-panel__percent"><span data-progress-percent>{{ run.progress_percent if run else 0 }}</span>%</div> <div class="progress-panel__meta">
</div> <span data-progress-status>{{ run.status_display if run else "Ожидание" }}</span>
</section> <span>обработано: <span data-progress-processed>{{ run.processed_count if run else 0 }}</span> / <span data-progress-found>{{ run.found_count if run else 0 }}</span></span>
<section class="panel"> <span>без изменений: <span data-progress-skipped>{{ run.skipped_count if run else 0 }}</span></span>
<h2 class="panel__title">Последние запуски</h2> <span>ошибок: <span data-progress-errors>{{ run.error_count if run else 0 }}</span></span>
<table class="table"> </div>
<thead><tr><th class="table__head">ID</th><th class="table__head">Статус</th><th class="table__head">Обработано</th><th class="table__head">Без изменений</th><th class="table__head">Ошибки</th><th class="table__head">Старт</th></tr></thead> <div class="progress-bar" aria-label="Parsing progress">
<tbody> <div class="progress-bar__fill" data-progress-fill style="width: {{ run.progress_percent if run else 0 }}%"></div>
{% for run in runs %} </div>
<tr class="table__row" onclick="window.location.href='/admin/runs/{{ run.id }}'" onkeydown="if (event.key === 'Enter' || event.key === ' ') { event.preventDefault(); window.location.href='/admin/runs/{{ run.id }}'; }" role="link" tabindex="0"><td class="table__cell">{{ run.id }}</td><td class="table__cell">{{ run.status_display }}</td><td class="table__cell">{{ run.parsed_count }}</td><td class="table__cell">{{ run.skipped_count }}</td><td class="table__cell">{{ run.error_count }}</td><td class="table__cell">{{ run.started_display }}</td></tr> <div class="progress-panel__percent"><span data-progress-percent>{{ run.progress_percent if run else 0 }}</span>%</div>
{% endfor %} </div>
</tbody> </section>
</table> <section class="panel">
</section> <h2 class="panel__title">Последние запуски</h2>
{% endblock %} <table class="table">
{% block scripts %} <thead><tr><th class="table__head">ID</th><th class="table__head">Статус</th><th class="table__head">Обработано</th><th class="table__head">Без изменений</th><th class="table__head">Ошибки</th><th class="table__head">Старт</th></tr></thead>
<script src="/static/admin.js"></script> <tbody>
{% endblock %} {% for run in runs %}
<tr class="table__row" onclick="window.location.href='/admin/runs/{{ run.id }}'" onkeydown="if (event.key === 'Enter' || event.key === ' ') { event.preventDefault(); window.location.href='/admin/runs/{{ run.id }}'; }" role="link" tabindex="0"><td class="table__cell">{{ run.id }}</td><td class="table__cell">{{ run.status_display }}</td><td class="table__cell">{{ run.parsed_count }}</td><td class="table__cell">{{ run.skipped_count }}</td><td class="table__cell">{{ run.error_count }}</td><td class="table__cell">{{ run.started_display }}</td></tr>
{% endfor %}
</tbody>
</table>
</section>
{% endblock %}
{% block scripts %}
<script src="/static/admin.js"></script>
{% endblock %}

6
app/version.py
View File

@@ -1,3 +1,3 @@
APP_VERSION = "0.7.0" APP_VERSION = "0.7.1"
FRONTEND_VERSION = "0.7.0" FRONTEND_VERSION = "0.7.1"
BACKEND_VERSION = "0.7.0" BACKEND_VERSION = "0.7.1"

56
pyproject.toml
View File

@@ -1,28 +1,28 @@
[project] [project]
name = "miem-workers" name = "miem-workers"
version = "0.7.0" version = "0.7.1"
description = "MIEM employees parser, admin API, and MCP server" description = "MIEM employees parser, admin API, and MCP server"
requires-python = ">=3.11" requires-python = ">=3.11"
dependencies = [ dependencies = [
"apscheduler>=3.10.4", "apscheduler>=3.10.4",
"beautifulsoup4>=4.12.3", "beautifulsoup4>=4.12.3",
"fastapi>=0.115.0", "fastapi>=0.115.0",
"httpx>=0.27.0", "httpx>=0.27.0",
"jinja2>=3.1.4", "jinja2>=3.1.4",
"lxml>=5.2.0", "lxml>=5.2.0",
"psycopg[binary]>=3.2.0", "psycopg[binary]>=3.2.0",
"pydantic-settings>=2.4.0", "pydantic-settings>=2.4.0",
"python-multipart>=0.0.9", "python-multipart>=0.0.9",
"requests>=2.32.0", "requests>=2.32.0",
"sqlalchemy>=2.0.32", "sqlalchemy>=2.0.32",
"uvicorn[standard]>=0.30.0", "uvicorn[standard]>=0.30.0",
] ]
[project.optional-dependencies] [project.optional-dependencies]
dev = [ dev = [
"pytest>=8.3.0", "pytest>=8.3.0",
] ]
[tool.pytest.ini_options] [tool.pytest.ini_options]
testpaths = ["tests"] testpaths = ["tests"]
pythonpath = ["."] pythonpath = ["."]

188
tests/test_admin_templates.py
View File

@@ -1,95 +1,105 @@
from pathlib import Path from pathlib import Path
def test_base_navigation_is_russian_and_has_no_legacy_employees_link(): def test_base_navigation_is_russian_and_has_no_legacy_employees_link():
template = Path("app/templates/base.html").read_text(encoding="utf-8") template = Path("app/templates/base.html").read_text(encoding="utf-8")
assert "Обзор" in template assert "Обзор" in template
assert "Сотрудники" in template assert "Сотрудники" in template
assert "Запуски" in template assert "Запуски" in template
assert "Выйти" in template assert "Выйти" in template
assert '<a class="admin__brand-link" href="/admin">MIEM Employees</a>' in template assert '<a class="admin__brand-link" href="/admin">MIEM Employees</a>' in template
assert ">Employees<" not in template assert ">Employees<" not in template
assert "/admin/employees" not in template assert "/admin/employees" not in template
def test_directory_template_is_russian_and_uses_display_dates(): def test_directory_template_is_russian_and_uses_display_dates():
template = Path("app/templates/directory.html").read_text(encoding="utf-8") template = Path("app/templates/directory.html").read_text(encoding="utf-8")
assert "Сотрудники" in template assert "Сотрудники" in template
assert "Колонки" in template assert "Колонки" in template
assert "Применить" in template assert "Применить" in template
assert "На странице: {{ value }}" in template assert "На странице: {{ value }}" in template
assert "{% for value in [25, 50, 100] %}" in template assert "{% for value in [25, 50, 100] %}" in template
assert "Найдено:" in template assert "Найдено:" in template
assert "Новости" in template assert "Новости" in template
assert "employee.news_count" in template assert "employee.news_count" in template
assert "employee.first_seen_display" in template assert "employee.first_seen_display" in template
assert "employee.last_seen_display" in template assert "employee.last_seen_display" in template
assert "employee.dismissed_display" in template assert "employee.dismissed_display" in template
assert "Directory" not in template assert "verification_required" in template
assert "employees found" not in template assert "Directory" not in template
assert "employees found" not in template
def test_admin_employees_route_redirects_to_directory():
source = Path("app/admin.py").read_text(encoding="utf-8") def test_admin_employees_route_redirects_to_directory():
source = Path("app/admin.py").read_text(encoding="utf-8")
assert 'RedirectResponse("/admin/directory", status_code=303)' in source
assert 'RedirectResponse("/admin/directory", status_code=303)' in source
def test_dashboard_limits_latest_runs_to_five():
source = Path("app/admin.py").read_text(encoding="utf-8") def test_dashboard_limits_latest_runs_to_five():
source = Path("app/admin.py").read_text(encoding="utf-8")
assert "order_by(desc(CrawlRun.started_at)).limit(5)" in source
assert "order_by(desc(CrawlRun.started_at)).limit(10)" not in source assert "order_by(desc(CrawlRun.started_at)).limit(5)" in source
assert "order_by(desc(CrawlRun.started_at)).limit(10)" not in source
def test_runs_template_links_to_run_detail():
template = Path("app/templates/runs.html").read_text(encoding="utf-8") def test_runs_template_links_to_run_detail():
template = Path("app/templates/runs.html").read_text(encoding="utf-8")
assert 'onclick="window.location.href=\'/admin/runs/{{ run.id }}\'"' in template
assert "onkeydown=\"if (event.key === 'Enter' || event.key === ' ')" in template assert 'onclick="window.location.href=\'/admin/runs/{{ run.id }}\'"' in template
assert 'role="link"' in template assert "onkeydown=\"if (event.key === 'Enter' || event.key === ' ')" in template
assert 'tabindex="0"' in template assert 'role="link"' in template
assert 'data-row-href="/admin/runs/{{ run.id }}"' not in template assert 'tabindex="0"' in template
assert '<a class="admin__link" href="/admin/runs/{{ run.id }}">' not in template assert 'data-row-href="/admin/runs/{{ run.id }}"' not in template
assert '<a class="admin__link" href="/admin/runs/{{ run.id }}">' not in template
def test_run_detail_template_extends_base_and_shows_change_groups():
template = Path("app/templates/run_detail.html").read_text(encoding="utf-8") def test_run_detail_template_extends_base_and_shows_change_groups():
template = Path("app/templates/run_detail.html").read_text(encoding="utf-8")
assert '{% extends "base.html" %}' in template
assert 'id="new-employees"' in template assert '{% extends "base.html" %}' in template
assert "Новые сотрудники" in template assert 'id="new-employees"' in template
assert "Потеряшки" in template assert "Новые сотрудники" in template
assert "Уволенные" in template assert "Потеряшки" in template
assert "Детализация сотрудников для этого запуска недоступна" in template assert "Требуют проверки" in template
assert "Уволенные" in template
assert "Детализация сотрудников для этого запуска недоступна" in template
def test_dashboard_metric_cards_link_to_admin_targets(): def test_dashboard_metric_cards_link_to_admin_targets():
template = Path("app/templates/dashboard.html").read_text(encoding="utf-8") template = Path("app/templates/dashboard.html").read_text(encoding="utf-8")
assert 'href="/admin/directory"' in template assert 'href="/admin/directory"' in template
assert 'href="/admin/directory?status=active"' in template assert 'href="/admin/directory?status=active"' in template
assert '/admin/runs/{{ latest_run.id }}#new-employees' in template assert 'href="/admin/directory?status=verification_required"' in template
assert 'href="/admin/directory?status=dismissed"' in template assert '/admin/runs/{{ latest_run.id }}#new-employees' in template
assert 'href="/admin/directory?status=dismissed"' in template
assert 'href="/admin/runs"' in template assert 'href="/admin/runs"' in template
def test_dashboard_latest_run_rows_link_to_run_detail(): def test_dashboard_has_dismissed_status_refresh_action():
template = Path("app/templates/dashboard.html").read_text(encoding="utf-8") template = Path("app/templates/dashboard.html").read_text(encoding="utf-8")
assert 'onclick="window.location.href=\'/admin/runs/{{ run.id }}\'"' in template assert 'action="/admin/dismissed/refresh"' in template
assert "onkeydown=\"if (event.key === 'Enter' || event.key === ' ')" in template assert "Проверить уволенных" in template
assert 'role="link"' in template
assert 'tabindex="0"' in template
assert 'data-row-href="/admin/runs/{{ run.id }}"' not in template def test_dashboard_latest_run_rows_link_to_run_detail():
assert '<a class="admin__link" href="/admin/runs/{{ run.id }}">' not in template template = Path("app/templates/dashboard.html").read_text(encoding="utf-8")
assert 'onclick="window.location.href=\'/admin/runs/{{ run.id }}\'"' in template
def test_admin_js_supports_keyboard_activation_for_clickable_rows(): assert "onkeydown=\"if (event.key === 'Enter' || event.key === ' ')" in template
source = Path("app/static/admin.js").read_text(encoding="utf-8") assert 'role="link"' in template
assert 'tabindex="0"' in template
assert 'addEventListener("keydown"' in source assert 'data-row-href="/admin/runs/{{ run.id }}"' not in template
assert '"Enter"' in source assert '<a class="admin__link" href="/admin/runs/{{ run.id }}">' not in template
assert '" "' in source
def test_admin_js_supports_keyboard_activation_for_clickable_rows():
source = Path("app/static/admin.js").read_text(encoding="utf-8")
assert 'addEventListener("keydown"' in source
assert '"Enter"' in source
assert '" "' in source

1064
tests/test_api_mcp.py
View File

File diff suppressed because it is too large Load Diff

924
tests/test_crawler.py
View File

@@ -1,383 +1,569 @@
import gzip import gzip
from datetime import datetime, timezone from datetime import datetime, timezone
from app.models import ( from app.models import (
CrawlError, CrawlError,
CrawlRun, CrawlRun,
CrawlRunEmployeeChange, CrawlRunEmployeeChange,
Employee, Employee,
EmployeeNewsLink, EmployeeNewsLink,
EmployeePublication, EmployeePublication,
EmployeeSnapshot, EmployeeSnapshot,
ParseResourceCache, ParseResourceCache,
) )
from app.services.crawler import _checksum, _mark_dismissed, _upsert_employee from app.config import Settings
from app.services.resource_cache import ResourceCache from app.services.crawler import _checksum, _mark_dismissed, _upsert_employee, refresh_dismissed_status
from app.services.resource_cache import ResourceCache
class FakeResponse:
def __init__(self, status_code): class FakeResponse:
self.status_code = status_code def __init__(self, status_code):
self.status_code = status_code
class FakeSession:
def __init__(self, statuses): class FakeSession:
self.statuses = statuses def __init__(self, statuses):
self.statuses = statuses
def get(self, url, **_kwargs):
return FakeResponse(self.statuses[url]) def get(self, url, **_kwargs):
return FakeResponse(self.statuses[url])
class ConditionalResponse:
def __init__(self, status_code, text="", headers=None): class ConditionalResponse:
self.status_code = status_code def __init__(self, status_code, text="", headers=None):
self._text = text self.status_code = status_code
self.headers = headers or {} self._text = text
self.text_read = False self.headers = headers or {}
self.text_read = False
@property
def text(self): @property
self.text_read = True def text(self):
return self._text self.text_read = True
return self._text
def raise_for_status(self):
return None def raise_for_status(self):
return None
class ConditionalSession: class ConditionalSession:
def __init__(self): def __init__(self):
self.requests = [] self.requests = []
self.not_modified_response = ConditionalResponse(304) self.not_modified_response = ConditionalResponse(304)
def get(self, url, **kwargs): def get(self, url, **kwargs):
self.requests.append((url, kwargs)) self.requests.append((url, kwargs))
if kwargs["headers"].get("If-None-Match") == '"cached"': if kwargs["headers"].get("If-None-Match") == '"cached"':
return self.not_modified_response return self.not_modified_response
return ConditionalResponse(200, "fresh", {"ETag": '"fresh"'}) return ConditionalResponse(200, "fresh", {"ETag": '"fresh"'})
def test_mark_dismissed_records_missing_source_when_profile_is_available(db_session): def test_refresh_dismissed_status_reactivates_only_profiles_in_source(monkeypatch, db_session):
run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") now = datetime.now(timezone.utc)
db_session.add(run) found = Employee(
db_session.add( profile_key="staff:returned",
Employee( canonical_url="https://www.hse.ru/staff/returned",
profile_key="staff:kept", status="dismissed",
canonical_url="https://www.hse.ru/staff/kept", dismissed_at=now,
status="active", first_seen_at=now,
first_seen_at=datetime.now(timezone.utc), last_seen_at=now,
last_seen_at=datetime.now(timezone.utc),
)
) )
db_session.add( still_dismissed = Employee(
Employee(
profile_key="staff:missing",
canonical_url="https://www.hse.ru/staff/missing",
status="active",
first_seen_at=datetime.now(timezone.utc),
last_seen_at=datetime.now(timezone.utc),
)
)
db_session.commit()
dismissed = _mark_dismissed(
db_session,
run,
{"staff:kept"},
FakeSession({"https://www.hse.ru/staff/missing": 200}),
30,
)
assert dismissed == 0
assert db_session.query(Employee).filter_by(profile_key="staff:kept").one().status == "active"
missing = db_session.query(Employee).filter_by(profile_key="staff:missing").one()
assert missing.status == "active"
assert missing.dismissed_at is None
change = db_session.query(CrawlRunEmployeeChange).one()
assert change.change_type == "missing_from_source"
assert change.profile_available is True
def test_mark_dismissed_marks_missing_employee_when_profile_is_unavailable(db_session):
run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
employee = Employee(
profile_key="staff:gone", profile_key="staff:gone",
canonical_url="https://www.hse.ru/staff/gone", canonical_url="https://www.hse.ru/staff/gone",
status="active", status="dismissed",
first_seen_at=datetime.now(timezone.utc), dismissed_at=now,
last_seen_at=datetime.now(timezone.utc), first_seen_at=now,
last_seen_at=now,
) )
db_session.add_all([run, employee]) db_session.add_all([found, still_dismissed])
db_session.commit() db_session.commit()
monkeypatch.setattr(
dismissed = _mark_dismissed( "app.services.crawler.collect_profile_links",
db_session, lambda *_args, **_kwargs: ["https://www.hse.ru/staff/returned"],
run,
set(),
FakeSession({"https://www.hse.ru/staff/gone": 404}),
30,
) )
assert dismissed == 1 run = refresh_dismissed_status(db_session, Settings())
assert employee.status == "dismissed"
assert employee.dismissed_at is not None
change = db_session.query(CrawlRunEmployeeChange).one()
assert change.change_type == "dismissed"
assert change.profile_available is False
assert run.status == "completed"
def test_upsert_employee_increments_new_count_and_records_change_for_new_employee(db_session): assert run.parsed_count == 1
run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") assert run.skipped_count == 1
db_session.add(run) assert found.status == "active"
db_session.commit() assert found.dismissed_at is None
assert still_dismissed.status == "dismissed"
_upsert_employee(
db_session,
run, def test_mark_dismissed_records_missing_source_when_profile_is_available(db_session):
{ run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
"source_url": "https://www.hse.ru/staff/newperson", db_session.add(run)
"profile_type": "staff", db_session.add(
"profile_id": "newperson", Employee(
"full_name": "New Person", profile_key="staff:kept",
"tabs": [], canonical_url="https://www.hse.ru/staff/kept",
"sections": [], status="active",
"parser_version": "0.2.0", first_seen_at=datetime.now(timezone.utc),
"_html": "<html></html>", last_seen_at=datetime.now(timezone.utc),
}, )
) )
db_session.commit() db_session.add(
Employee(
assert run.new_count == 1 profile_key="staff:missing",
change = db_session.query(CrawlRunEmployeeChange).one() canonical_url="https://www.hse.ru/staff/missing",
assert change.change_type == "new" status="active",
assert change.full_name == "New Person" first_seen_at=datetime.now(timezone.utc),
last_seen_at=datetime.now(timezone.utc),
)
def test_resource_cache_uses_etag_and_reuses_cached_body_on_304(db_session): )
db_session.add( db_session.commit()
ParseResourceCache(
profile_key="staff:cached", dismissed = _mark_dismissed(
resource_key="main-html", db_session,
method="GET", run,
url="https://www.hse.ru/staff/cached", {"staff:kept"},
request_fingerprint="020d59db7b358d9023d0f185bcbf5a9c085d3cf2bf91d92d48eee9147e8d0f01", FakeSession({"https://www.hse.ru/staff/missing": 200}),
etag='"cached"', 30,
body_hash="cached-hash", )
body_snapshot=gzip.compress("cached body".encode("utf-8")),
parser_version="0.6.0", assert dismissed == 0
) assert db_session.query(Employee).filter_by(profile_key="staff:kept").one().status == "active"
) missing = db_session.query(Employee).filter_by(profile_key="staff:missing").one()
db_session.commit() assert missing.status == "active"
session = ConditionalSession() assert missing.dismissed_at is None
change = db_session.query(CrawlRunEmployeeChange).one()
result = ResourceCache(db_session).fetch_text( assert change.change_type == "missing_from_source"
session, assert change.profile_available is True
profile_key="staff:cached",
resource_key="main-html",
method="GET", def test_mark_dismissed_requires_consecutive_unavailable_checks(db_session):
url="https://www.hse.ru/staff/cached", run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
headers={"User-Agent": "test"}, employee = Employee(
timeout=10, profile_key="staff:gone",
) canonical_url="https://www.hse.ru/staff/gone",
status="active",
assert session.requests[0][1]["headers"]["If-None-Match"] == '"cached"' first_seen_at=datetime.now(timezone.utc),
assert result.text == "cached body" last_seen_at=datetime.now(timezone.utc),
assert result.from_cache is True )
assert session.not_modified_response.text_read is False db_session.add_all([run, employee])
db_session.commit()
def test_upsert_employee_skips_snapshot_when_checksum_is_unchanged(db_session): first_check = _mark_dismissed(
first_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") db_session,
second_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") run,
db_session.add_all([first_run, second_run]) set(),
db_session.commit() FakeSession({"https://www.hse.ru/staff/gone": 404}),
30,
_, first_changed = _upsert_employee(db_session, first_run, _parsed_employee("same")) confirmation_runs=2,
_, second_changed = _upsert_employee(db_session, second_run, _parsed_employee("same")) )
db_session.commit()
assert first_check == 0
assert first_changed is True assert employee.status == "verification_required"
assert second_changed is False assert employee.dismissed_at is None
assert db_session.query(EmployeeSnapshot).count() == 1 assert employee.profile_unavailable_streak == 1
assert db_session.query(CrawlRunEmployeeChange).one().change_type == "verification_required"
def test_upsert_employee_saves_publications_and_reuses_existing_rows(db_session): second_check = _mark_dismissed(
first_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") db_session,
second_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") run,
db_session.add_all([first_run, second_run]) set(),
db_session.commit() FakeSession({"https://www.hse.ru/staff/gone": 404}),
30,
parsed = _parsed_employee("published") confirmation_runs=2,
parsed["sections"] = [ )
{
"type": "publications", assert second_check == 1
"publications": [ assert employee.status == "dismissed"
{ assert employee.dismissed_at is not None
"id": "888959076", assert employee.profile_unavailable_streak == 2
"publication_id": "888959076", change = db_session.query(CrawlRunEmployeeChange).order_by(CrawlRunEmployeeChange.id).all()[-1]
"title": "Detailed Publication", assert change.change_type == "dismissed"
"year": 2023, assert change.profile_available is False
"publication_type": "ARTICLE",
"language": "ru",
"status": 1, def test_mark_dismissed_does_not_dismiss_on_server_error(db_session):
"url": "https://publications.hse.ru/view/888959076", run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
"doi_url": "https://doi.org/10.1/test", employee = Employee(
"citation_text": "Detailed citation", profile_key="staff:temporary-error",
"annotation": {"ru": "Аннотация"}, canonical_url="https://www.hse.ru/staff/temporary-error",
"description": {"main": "Detailed citation"}, status="active",
"authors": [{"id": "1", "title_ru": "Автор"}], first_seen_at=datetime.now(timezone.utc),
"raw_data": {"id": "888959076", "title": "Detailed Publication"}, last_seen_at=datetime.now(timezone.utc),
} )
], db_session.add_all([run, employee])
} db_session.commit()
]
dismissed = _mark_dismissed(
employee, _ = _upsert_employee(db_session, first_run, parsed) db_session,
db_session.commit() run,
_upsert_employee(db_session, second_run, _parsed_employee_with_publication("published")) set(),
db_session.commit() FakeSession({"https://www.hse.ru/staff/temporary-error": 503}),
30,
publications = db_session.query(EmployeePublication).filter_by(employee_id=employee.id).all() confirmation_runs=1,
assert len(publications) == 1 )
assert publications[0].doi_url == "https://doi.org/10.1/test"
assert publications[0].authors == [{"id": "1", "title_ru": "Автор"}] assert dismissed == 0
assert employee.status == "active"
assert employee.profile_unavailable_streak == 0
def test_upsert_employee_records_publication_errors_without_failing_employee(monkeypatch, db_session): assert db_session.query(CrawlError).one().error_type == "ProfileAvailabilityCheckError"
run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
db_session.add(run)
db_session.commit() def test_available_profile_resets_verification_streak(db_session):
run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
def broken_sync(*_args, **_kwargs): employee = Employee(
raise RuntimeError("boom") profile_key="staff:restored",
canonical_url="https://www.hse.ru/staff/restored",
monkeypatch.setattr("app.services.crawler._sync_employee_publications", broken_sync) status="active",
first_seen_at=datetime.now(timezone.utc),
employee, changed = _upsert_employee(db_session, run, _parsed_employee_with_publication("error-safe")) last_seen_at=datetime.now(timezone.utc),
db_session.commit() )
db_session.add_all([run, employee])
assert changed is True db_session.commit()
assert employee.full_name == "Same Person"
assert db_session.query(Employee).filter_by(profile_key="staff:error-safe").one() _mark_dismissed(
error = db_session.query(CrawlError).one() db_session,
assert "публикации" in error.message.lower() run,
set(),
FakeSession({"https://www.hse.ru/staff/restored": 404}),
def test_upsert_employee_saves_news_links_and_reuses_existing_rows(db_session): 30,
first_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") confirmation_runs=3,
second_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") )
db_session.add_all([first_run, second_run]) _mark_dismissed(
db_session.commit() db_session,
run,
employee, _ = _upsert_employee(db_session, first_run, _parsed_employee_with_news("news-person")) set(),
db_session.commit() FakeSession({"https://www.hse.ru/staff/restored": 200}),
_upsert_employee(db_session, second_run, _parsed_employee_with_news("news-person")) 30,
db_session.commit() confirmation_runs=3,
)
news_links = db_session.query(EmployeeNewsLink).filter_by(employee_id=employee.id).all()
assert len(news_links) == 1 assert employee.status == "active"
assert news_links[0].title == "News Title" assert employee.profile_unavailable_streak == 0
assert news_links[0].url == "https://www.hse.ru/news/1.html"
assert news_links[0].published_year == 2026
def test_mark_dismissed_blocks_mass_auto_dismissals(db_session):
run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
def test_upsert_employee_records_news_errors_without_failing_employee(monkeypatch, db_session): employees = [
run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") Employee(
db_session.add(run) profile_key=f"staff:gone-{index}",
db_session.commit() canonical_url=f"https://www.hse.ru/staff/gone-{index}",
status="active",
def broken_sync(*_args, **_kwargs): first_seen_at=datetime.now(timezone.utc),
raise RuntimeError("boom") last_seen_at=datetime.now(timezone.utc),
)
monkeypatch.setattr("app.services.crawler._sync_employee_news_links", broken_sync) for index in range(2)
]
employee, changed = _upsert_employee(db_session, run, _parsed_employee_with_news("news-error-safe")) db_session.add_all([run, *employees])
db_session.commit() db_session.commit()
assert changed is True dismissed = _mark_dismissed(
assert employee.full_name == "Same Person" db_session,
assert db_session.query(Employee).filter_by(profile_key="staff:news-error-safe").one() run,
error = db_session.query(CrawlError).one() set(),
assert "новости" in error.message.lower() FakeSession({employee.canonical_url: 404 for employee in employees}),
30,
confirmation_runs=1,
def test_checksum_changes_when_widget_data_changes(): max_auto_dismissals=1,
base = _parsed_employee("widgets") )
changed = _parsed_employee("widgets")
changed["sections"] = [ assert dismissed == 0
{ assert {employee.status for employee in employees} == {"verification_required"}
"type": "publications", assert "приостановлено" in run.message
"publications": [{"id": "1", "title": "New publication"}],
}
] def test_upsert_employee_increments_new_count_and_records_change_for_new_employee(db_session):
run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
assert _checksum(base) != _checksum(changed) db_session.add(run)
db_session.commit()
def test_checksum_ignores_date_dependent_experience_text(): _upsert_employee(
first = _parsed_employee("experience") db_session,
second = _parsed_employee("experience") run,
first["sections"] = [{"raw_text": "Стаж работы в НИУ ВШЭ: 5 лет"}] {
second["sections"] = [{"raw_text": "Стаж работы в НИУ ВШЭ: 6 лет"}] "source_url": "https://www.hse.ru/staff/newperson",
"profile_type": "staff",
assert _checksum(first) == _checksum(second) "profile_id": "newperson",
"full_name": "New Person",
"tabs": [],
def _parsed_employee(profile_id: str) -> dict: "sections": [],
return { "parser_version": "0.2.0",
"source_url": f"https://www.hse.ru/staff/{profile_id}", "_html": "<html></html>",
"profile_type": "staff", },
"profile_id": profile_id, )
"full_name": "Same Person", db_session.commit()
"tabs": [],
"sections": [], assert run.new_count == 1
"parser_version": "0.6.0", change = db_session.query(CrawlRunEmployeeChange).one()
"_html": "<html></html>", assert change.change_type == "new"
} assert change.full_name == "New Person"
def _parsed_employee_with_publication(profile_id: str) -> dict: def test_upsert_employee_reconciles_profile_moved_to_new_url(db_session):
parsed = _parsed_employee(profile_id) run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
parsed["sections"] = [ employee = Employee(
{ profile_key="staff:abelov",
"type": "publications", canonical_url="https://www.hse.ru/staff/abelov",
"publications": [ full_name="Белов Александр Владимирович",
{ status="active",
"id": "888959076", first_seen_at=datetime.now(timezone.utc),
"publication_id": "888959076", last_seen_at=datetime.now(timezone.utc),
"title": "Detailed Publication", )
"year": 2023, db_session.add_all([run, employee])
"publication_type": "ARTICLE", db_session.commit()
"language": "ru", employee_id = employee.id
"status": 1,
"url": "https://publications.hse.ru/view/888959076", updated, changed = _upsert_employee(
"doi_url": "https://doi.org/10.1/test", db_session,
"citation_text": "Detailed citation", run,
"annotation": {"ru": "Аннотация"}, {
"description": {"main": "Detailed citation"}, "source_url": "https://www.hse.ru/org/persons/47634735",
"authors": [{"id": "1", "title_ru": "Автор"}], "profile_type": "org_person",
"raw_data": {"id": "888959076", "title": "Detailed Publication"}, "profile_id": "47634735",
} "full_name": "Белов Александр Владимирович",
], "tabs": [],
} "sections": [],
] "parser_version": "0.7.0",
return parsed "_html": "<html></html>",
},
)
def _parsed_employee_with_news(profile_id: str) -> dict: db_session.commit()
parsed = _parsed_employee(profile_id)
parsed["sections"] = [ assert changed is True
{ assert updated.id == employee_id
"type": "news", assert updated.profile_key == "org_person:47634735"
"news_links": [ assert updated.canonical_url == "https://www.hse.ru/org/persons/47634735"
{ assert updated.status == "active"
"title": "News Title", assert run.new_count == 0
"url": "https://www.hse.ru/news/1.html", assert db_session.query(Employee).count() == 1
"summary": "News summary", assert {item.url for item in updated.profile_urls} == {
"published_at": "2026-04-28T00:00:00+00:00", "https://www.hse.ru/staff/abelov",
"published_year": 2026, "https://www.hse.ru/org/persons/47634735",
"raw_data": {"title": "News Title", "url": "https://www.hse.ru/news/1.html"}, }
}
],
} def test_resource_cache_uses_etag_and_reuses_cached_body_on_304(db_session):
] db_session.add(
return parsed ParseResourceCache(
profile_key="staff:cached",
resource_key="main-html",
method="GET",
url="https://www.hse.ru/staff/cached",
request_fingerprint="020d59db7b358d9023d0f185bcbf5a9c085d3cf2bf91d92d48eee9147e8d0f01",
etag='"cached"',
body_hash="cached-hash",
body_snapshot=gzip.compress("cached body".encode("utf-8")),
parser_version="0.6.0",
)
)
db_session.commit()
session = ConditionalSession()
result = ResourceCache(db_session).fetch_text(
session,
profile_key="staff:cached",
resource_key="main-html",
method="GET",
url="https://www.hse.ru/staff/cached",
headers={"User-Agent": "test"},
timeout=10,
)
assert session.requests[0][1]["headers"]["If-None-Match"] == '"cached"'
assert result.text == "cached body"
assert result.from_cache is True
assert session.not_modified_response.text_read is False
def test_upsert_employee_skips_snapshot_when_checksum_is_unchanged(db_session):
first_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
second_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
db_session.add_all([first_run, second_run])
db_session.commit()
_, first_changed = _upsert_employee(db_session, first_run, _parsed_employee("same"))
_, second_changed = _upsert_employee(db_session, second_run, _parsed_employee("same"))
db_session.commit()
assert first_changed is True
assert second_changed is False
assert db_session.query(EmployeeSnapshot).count() == 1
def test_upsert_employee_saves_publications_and_reuses_existing_rows(db_session):
first_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
second_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
db_session.add_all([first_run, second_run])
db_session.commit()
parsed = _parsed_employee("published")
parsed["sections"] = [
{
"type": "publications",
"publications": [
{
"id": "888959076",
"publication_id": "888959076",
"title": "Detailed Publication",
"year": 2023,
"publication_type": "ARTICLE",
"language": "ru",
"status": 1,
"url": "https://publications.hse.ru/view/888959076",
"doi_url": "https://doi.org/10.1/test",
"citation_text": "Detailed citation",
"annotation": {"ru": "Аннотация"},
"description": {"main": "Detailed citation"},
"authors": [{"id": "1", "title_ru": "Автор"}],
"raw_data": {"id": "888959076", "title": "Detailed Publication"},
}
],
}
]
employee, _ = _upsert_employee(db_session, first_run, parsed)
db_session.commit()
_upsert_employee(db_session, second_run, _parsed_employee_with_publication("published"))
db_session.commit()
publications = db_session.query(EmployeePublication).filter_by(employee_id=employee.id).all()
assert len(publications) == 1
assert publications[0].doi_url == "https://doi.org/10.1/test"
assert publications[0].authors == [{"id": "1", "title_ru": "Автор"}]
def test_upsert_employee_records_publication_errors_without_failing_employee(monkeypatch, db_session):
run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
db_session.add(run)
db_session.commit()
def broken_sync(*_args, **_kwargs):
raise RuntimeError("boom")
monkeypatch.setattr("app.services.crawler._sync_employee_publications", broken_sync)
employee, changed = _upsert_employee(db_session, run, _parsed_employee_with_publication("error-safe"))
db_session.commit()
assert changed is True
assert employee.full_name == "Same Person"
assert db_session.query(Employee).filter_by(profile_key="staff:error-safe").one()
error = db_session.query(CrawlError).one()
assert "публикации" in error.message.lower()
def test_upsert_employee_saves_news_links_and_reuses_existing_rows(db_session):
first_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
second_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
db_session.add_all([first_run, second_run])
db_session.commit()
employee, _ = _upsert_employee(db_session, first_run, _parsed_employee_with_news("news-person"))
db_session.commit()
_upsert_employee(db_session, second_run, _parsed_employee_with_news("news-person"))
db_session.commit()
news_links = db_session.query(EmployeeNewsLink).filter_by(employee_id=employee.id).all()
assert len(news_links) == 1
assert news_links[0].title == "News Title"
assert news_links[0].url == "https://www.hse.ru/news/1.html"
assert news_links[0].published_year == 2026
def test_upsert_employee_records_news_errors_without_failing_employee(monkeypatch, db_session):
run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
db_session.add(run)
db_session.commit()
def broken_sync(*_args, **_kwargs):
raise RuntimeError("boom")
monkeypatch.setattr("app.services.crawler._sync_employee_news_links", broken_sync)
employee, changed = _upsert_employee(db_session, run, _parsed_employee_with_news("news-error-safe"))
db_session.commit()
assert changed is True
assert employee.full_name == "Same Person"
assert db_session.query(Employee).filter_by(profile_key="staff:news-error-safe").one()
error = db_session.query(CrawlError).one()
assert "новости" in error.message.lower()
def test_checksum_changes_when_widget_data_changes():
base = _parsed_employee("widgets")
changed = _parsed_employee("widgets")
changed["sections"] = [
{
"type": "publications",
"publications": [{"id": "1", "title": "New publication"}],
}
]
assert _checksum(base) != _checksum(changed)
def test_checksum_ignores_date_dependent_experience_text():
first = _parsed_employee("experience")
second = _parsed_employee("experience")
first["sections"] = [{"raw_text": "Стаж работы в НИУ ВШЭ: 5 лет"}]
second["sections"] = [{"raw_text": "Стаж работы в НИУ ВШЭ: 6 лет"}]
assert _checksum(first) == _checksum(second)
def _parsed_employee(profile_id: str) -> dict:
return {
"source_url": f"https://www.hse.ru/staff/{profile_id}",
"profile_type": "staff",
"profile_id": profile_id,
"full_name": "Same Person",
"tabs": [],
"sections": [],
"parser_version": "0.6.0",
"_html": "<html></html>",
}
def _parsed_employee_with_publication(profile_id: str) -> dict:
parsed = _parsed_employee(profile_id)
parsed["sections"] = [
{
"type": "publications",
"publications": [
{
"id": "888959076",
"publication_id": "888959076",
"title": "Detailed Publication",
"year": 2023,
"publication_type": "ARTICLE",
"language": "ru",
"status": 1,
"url": "https://publications.hse.ru/view/888959076",
"doi_url": "https://doi.org/10.1/test",
"citation_text": "Detailed citation",
"annotation": {"ru": "Аннотация"},
"description": {"main": "Detailed citation"},
"authors": [{"id": "1", "title_ru": "Автор"}],
"raw_data": {"id": "888959076", "title": "Detailed Publication"},
}
],
}
]
return parsed
def _parsed_employee_with_news(profile_id: str) -> dict:
parsed = _parsed_employee(profile_id)
parsed["sections"] = [
{
"type": "news",
"news_links": [
{
"title": "News Title",
"url": "https://www.hse.ru/news/1.html",
"summary": "News summary",
"published_at": "2026-04-28T00:00:00+00:00",
"published_year": 2026,
"raw_data": {"title": "News Title", "url": "https://www.hse.ru/news/1.html"},
}
],
}
]
return parsed