feat: add dismissed employee status refresh #29
5
CHANGELOG.md
Normal file
5
CHANGELOG.md
Normal file
@@ -0,0 +1,5 @@
|
||||
# Changelog
|
||||
|
||||
## 0.7.1
|
||||
|
||||
- Добавлена кнопка «Проверить уволенных» для принудительной сверки статуса уволенных сотрудников с текущим списком источника.
|
||||
290
README.md
290
README.md
@@ -1,142 +1,150 @@
|
||||
# MIEM Employees Server
|
||||
|
||||
Сервис собирает сотрудников МИЭМ с сайта ВШЭ, хранит карточки и историю обновлений в Postgres, показывает минимальную админку и отдает read-only MCP endpoint для ИИ-агентов.
|
||||
|
||||
## Архитектура
|
||||
|
||||
- `api`: FastAPI, REST API, HTML-админка, healthcheck.
|
||||
- `worker`: weekly scheduler, который запускает парсинг по `CRAWL_CRON`.
|
||||
- `mcp`: открытый HTTP MCP endpoint для ИИ-агентов.
|
||||
- `postgres`: основная БД.
|
||||
|
||||
Парсер использует фиксированный источник сотрудников, по умолчанию `https://miem.hse.ru/persons`. Для каждой карточки сохраняются ФИО, должности, год начала работы, контакты, идентификаторы, вкладки профиля, секции, публикации, курсы, ВКР, новости, JSON-снапшот и сжатый HTML-снапшот. Детальные публикации дополнительно нормализуются в отдельную таблицу `employee_publications`, а новости из блока «В новостях» — в `employee_news_links`. Ссылки обходятся только из меню профиля самого сотрудника (`person-menu`), например `#sci`, `#teaching`, `#main`.
|
||||
|
||||
## Переменные окружения
|
||||
|
||||
Скопируйте `.env.example` в `.env` и поменяйте секреты:
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
Основные настройки:
|
||||
|
||||
- `DATABASE_URL`: строка подключения SQLAlchemy.
|
||||
- `SOURCE_URL`: список сотрудников МИЭМ.
|
||||
- `CRAWL_CRON`: расписание в формате crontab, по умолчанию `0 3 * * 1`.
|
||||
- `CRAWL_LIMIT`: опциональный лимит профилей для тестового запуска.
|
||||
- `ADMIN_USERNAME`, `ADMIN_PASSWORD`: логин и пароль админки.
|
||||
- `SESSION_SECRET`: секрет подписи cookie.
|
||||
- `PARSER_USE_PLAYWRIGHT`: включение Playwright-рендера динамических вкладок.
|
||||
|
||||
## Локальный запуск
|
||||
|
||||
```bash
|
||||
python -m venv .venv
|
||||
.venv\Scripts\activate
|
||||
pip install -r requirements.txt
|
||||
uvicorn app.main:app --reload
|
||||
```
|
||||
|
||||
Админка: `http://localhost:8000/admin`.
|
||||
|
||||
В админке доступны:
|
||||
|
||||
- `Dashboard`: общая статистика, последний добавленный сотрудник, прогресс текущего/последнего парсинга и ручной запуск.
|
||||
- `Directory`: настраиваемая таблица сотрудников с фильтрами, сортировкой, пагинацией и выбором колонок.
|
||||
- `Runs`: история запусков, ошибки и progress bar.
|
||||
|
||||
## Docker Compose
|
||||
|
||||
```bash
|
||||
docker compose up --build
|
||||
```
|
||||
|
||||
По умолчанию:
|
||||
|
||||
- API и админка: `http://localhost:8000`
|
||||
- MCP: `http://localhost:8001/mcp`
|
||||
- Postgres: `localhost:5432`
|
||||
|
||||
Таблицы создаются приложением при старте. При обновлении существующей базы приложение также добавляет недостающие runtime-колонки, например `crawl_runs.skipped_count`. SQL-миграции для ручного применения лежат в `migrations/`.
|
||||
|
||||
## Наполнение БД
|
||||
|
||||
Основная карточка сотрудника хранится в `employees`: профиль, статус, даты обнаружения/увольнения, текущий JSON `current_data`, checksum и версия парсера. История успешных изменений сохраняется в `employee_snapshots` вместе с JSON-снимком и сжатым HTML профиля.
|
||||
|
||||
Публикации теперь хранятся в двух видах:
|
||||
|
||||
- краткий список остается внутри `employees.current_data.sections[].publications` для обратной совместимости;
|
||||
- детальные записи сохраняются в `employee_publications` и связываются с сотрудником через `employee_id`.
|
||||
|
||||
`employee_publications` содержит `publication_id`, название, год, тип публикации, язык, статус, ссылку на карточку HSE Publications, DOI, внешние/document-ссылки, citation text, аннотацию, описание, авторов, raw JSON ответа `searchPubs` и `source_hash` для безопасного повторного upsert. Уникальность поддерживается по `(employee_id, publication_id)` и `(employee_id, source_hash)`, поэтому повторный crawl не должен создавать дубликаты.
|
||||
|
||||
`list_employee_publications` сначала читает `employee_publications`; если детальных строк еще нет, возвращает старые публикации из `current_data`.
|
||||
|
||||
Новости сотрудников также хранятся в двух видах:
|
||||
|
||||
- краткий список остается внутри `employees.current_data.sections[].news_links`;
|
||||
- нормализованные карточки из вкладки «В новостях» сохраняются в `employee_news_links`.
|
||||
|
||||
`employee_news_links` содержит название новости, ссылку, краткое описание, дату публикации, год публикации, raw JSON карточки и `source_hash`. Уникальность поддерживается по `(employee_id, url)` и `(employee_id, source_hash)`, поэтому повторный crawl не создает дубликаты.
|
||||
|
||||
## Парсинг
|
||||
|
||||
Weekly worker запускается по `CRAWL_CRON`. Ручной запуск доступен в админке на `Dashboard` и странице `Runs` или через REST:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8000/api/crawl-runs --cookie "miem_admin_session=..."
|
||||
```
|
||||
|
||||
Алгоритм обновления:
|
||||
|
||||
- найденные сотрудники получают статус `active` и обновленный `last_seen_at`;
|
||||
- новые сотрудники добавляются в `employees`;
|
||||
- количество новых сотрудников за запуск сохраняется в `crawl_runs.new_count`;
|
||||
- публикации из HSE Publications записываются в `employee_publications`, а краткий список остается в JSON профиля;
|
||||
- новости из блока «В новостях» записываются в `employee_news_links`, а краткий список остается в JSON профиля;
|
||||
- активные сотрудники, исчезнувшие из текущего списка источника, получают статус `dismissed` и `dismissed_at`;
|
||||
# MIEM Employees Server
|
||||
|
||||
Сервис собирает сотрудников МИЭМ с сайта ВШЭ, хранит карточки и историю обновлений в Postgres, показывает минимальную админку и отдает read-only MCP endpoint для ИИ-агентов.
|
||||
|
||||
## Архитектура
|
||||
|
||||
- `api`: FastAPI, REST API, HTML-админка, healthcheck.
|
||||
- `worker`: weekly scheduler, который запускает парсинг по `CRAWL_CRON`.
|
||||
- `mcp`: открытый HTTP MCP endpoint для ИИ-агентов.
|
||||
- `postgres`: основная БД.
|
||||
|
||||
Парсер использует фиксированный источник сотрудников, по умолчанию `https://miem.hse.ru/persons`. Для каждой карточки сохраняются ФИО, должности, год начала работы, контакты, идентификаторы, вкладки профиля, секции, публикации, курсы, ВКР, новости, JSON-снапшот и сжатый HTML-снапшот. Детальные публикации дополнительно нормализуются в отдельную таблицу `employee_publications`, а новости из блока «В новостях» — в `employee_news_links`. Ссылки обходятся только из меню профиля самого сотрудника (`person-menu`), например `#sci`, `#teaching`, `#main`.
|
||||
|
||||
## Переменные окружения
|
||||
|
||||
Скопируйте `.env.example` в `.env` и поменяйте секреты:
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
Основные настройки:
|
||||
|
||||
- `DATABASE_URL`: строка подключения SQLAlchemy.
|
||||
- `SOURCE_URL`: список сотрудников МИЭМ.
|
||||
- `CRAWL_CRON`: расписание в формате crontab, по умолчанию `0 3 * * 1`.
|
||||
- `CRAWL_LIMIT`: опциональный лимит профилей для тестового запуска.
|
||||
- `ADMIN_USERNAME`, `ADMIN_PASSWORD`: логин и пароль админки.
|
||||
- `SESSION_SECRET`: секрет подписи cookie.
|
||||
- `PARSER_USE_PLAYWRIGHT`: включение Playwright-рендера динамических вкладок.
|
||||
- `DISMISSAL_CONFIRMATION_RUNS`: сколько последовательных проверок недоступности нужно для увольнения, по умолчанию `3`.
|
||||
- `MAX_AUTO_DISMISSALS_PER_RUN`: защитный лимит массовых автоматических увольнений за один запуск, по умолчанию `25`.
|
||||
|
||||
## Локальный запуск
|
||||
|
||||
```bash
|
||||
python -m venv .venv
|
||||
.venv\Scripts\activate
|
||||
pip install -r requirements.txt
|
||||
uvicorn app.main:app --reload
|
||||
```
|
||||
|
||||
Админка: `http://localhost:8000/admin`.
|
||||
|
||||
В админке доступны:
|
||||
|
||||
- `Dashboard`: общая статистика, последний добавленный сотрудник, прогресс текущего/последнего парсинга и ручной запуск.
|
||||
- `Directory`: настраиваемая таблица сотрудников с фильтрами, сортировкой, пагинацией и выбором колонок.
|
||||
- `Runs`: история запусков, ошибки и progress bar.
|
||||
|
||||
## Docker Compose
|
||||
|
||||
```bash
|
||||
docker compose up --build
|
||||
```
|
||||
|
||||
По умолчанию:
|
||||
|
||||
- API и админка: `http://localhost:8000`
|
||||
- MCP: `http://localhost:8001/mcp`
|
||||
- Postgres: `localhost:5432`
|
||||
|
||||
Таблицы создаются приложением при старте. При обновлении существующей базы приложение также добавляет недостающие runtime-колонки, например `crawl_runs.skipped_count`. SQL-миграции для ручного применения лежат в `migrations/`.
|
||||
|
||||
## Наполнение БД
|
||||
|
||||
Основная карточка сотрудника хранится в `employees`: профиль, статус, даты обнаружения/увольнения, текущий JSON `current_data`, checksum и версия парсера. История успешных изменений сохраняется в `employee_snapshots` вместе с JSON-снимком и сжатым HTML профиля.
|
||||
|
||||
Публикации теперь хранятся в двух видах:
|
||||
|
||||
- краткий список остается внутри `employees.current_data.sections[].publications` для обратной совместимости;
|
||||
- детальные записи сохраняются в `employee_publications` и связываются с сотрудником через `employee_id`.
|
||||
|
||||
`employee_publications` содержит `publication_id`, название, год, тип публикации, язык, статус, ссылку на карточку HSE Publications, DOI, внешние/document-ссылки, citation text, аннотацию, описание, авторов, raw JSON ответа `searchPubs` и `source_hash` для безопасного повторного upsert. Уникальность поддерживается по `(employee_id, publication_id)` и `(employee_id, source_hash)`, поэтому повторный crawl не должен создавать дубликаты.
|
||||
|
||||
`list_employee_publications` сначала читает `employee_publications`; если детальных строк еще нет, возвращает старые публикации из `current_data`.
|
||||
|
||||
Новости сотрудников также хранятся в двух видах:
|
||||
|
||||
- краткий список остается внутри `employees.current_data.sections[].news_links`;
|
||||
- нормализованные карточки из вкладки «В новостях» сохраняются в `employee_news_links`.
|
||||
|
||||
`employee_news_links` содержит название новости, ссылку, краткое описание, дату публикации, год публикации, raw JSON карточки и `source_hash`. Уникальность поддерживается по `(employee_id, url)` и `(employee_id, source_hash)`, поэтому повторный crawl не создает дубликаты.
|
||||
|
||||
## Парсинг
|
||||
|
||||
Weekly worker запускается по `CRAWL_CRON`. Ручной запуск доступен в админке на `Dashboard` и странице `Runs` или через REST:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8000/api/crawl-runs --cookie "miem_admin_session=..."
|
||||
```
|
||||
|
||||
Алгоритм обновления:
|
||||
|
||||
- найденные сотрудники получают статус `active` и обновленный `last_seen_at`;
|
||||
- новые сотрудники добавляются в `employees`;
|
||||
- если профиль перенесен на другой URL, он сопоставляется с прежней записью по единственному точному совпадению ФИО;
|
||||
- старые URL сохраняются в истории `employee_profile_urls`;
|
||||
- количество новых сотрудников за запуск сохраняется в `crawl_runs.new_count`;
|
||||
- публикации из HSE Publications записываются в `employee_publications`, а краткий список остается в JSON профиля;
|
||||
- новости из блока «В новостях» записываются в `employee_news_links`, а краткий список остается в JSON профиля;
|
||||
- один `404` старого профиля переводит сотрудника в статус `verification_required`, а не в `dismissed`;
|
||||
- статус `dismissed` устанавливается только после нескольких последовательных проверок `404`/`410`;
|
||||
- сетевые ошибки и ответы `5xx` не считаются подтверждением увольнения;
|
||||
- если число кандидатов на увольнение превышает защитный лимит, автоматическое увольнение приостанавливается;
|
||||
- кнопка «Проверить уволенных» сверяет только их profile_key с текущим списком источника и возвращает найденных сотрудников в `active` без обновления содержимого профиля;
|
||||
- каждый успешный новый или измененный разбор сохраняет запись в `employee_snapshots`;
|
||||
- неизмененные профили учитываются в `crawl_runs.skipped_count` и не получают новый snapshot.
|
||||
|
||||
Во время выполнения парсинга `found_count`, `parsed_count`, `skipped_count` и `error_count` обновляются в базе. Админка опрашивает `/api/crawl-runs/latest` и показывает прогресс как `(parsed_count + skipped_count + error_count) / found_count`.
|
||||
|
||||
## MCP
|
||||
|
||||
Endpoint: `POST /mcp`, без авторизации на уровне приложения.
|
||||
|
||||
Поддерживаемые tools:
|
||||
|
||||
- `get_service_info()`
|
||||
- `sync_employees(client_hash?, include_data?)`
|
||||
- `search_employees(query, status?, limit?)`
|
||||
- `get_employee(profile_id_or_url)`
|
||||
- `list_employee_publications(profile_id_or_url)` — публикации сотрудника; при наличии данных из `employee_publications` возвращает авторов, DOI, аннотацию, описание, citation text, год, тип, язык, статус и ссылку HSE Publications.
|
||||
- `list_employee_courses(profile_id_or_url)`
|
||||
- `get_crawl_status()`
|
||||
- `get_crawl_run_details(run_id)`
|
||||
|
||||
`get_service_info` возвращает метаданные сервиса, список tools и текущую версию набора сотрудников. `sync_employees` отдает полный snapshot или delta по `client_hash`; checksum набора строится по сотрудникам, их статусам и текущим checksums. Ответы tools возвращаются как JSON-строка внутри MCP `content[0].text`.
|
||||
|
||||
Новости сотрудника отдельной MCP tool не имеют: они доступны в `get_employee(...).data.sections` и `sync_employees(include_data=true)` как секция `type = "news"` с массивом `news_links`.
|
||||
|
||||
Пример локального запроса списка tools:
|
||||
|
||||
```bash
|
||||
curl http://localhost:8001/mcp \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}'
|
||||
```
|
||||
|
||||
Если MCP нужно ограничить, делайте это на сетевом уровне: localhost binding, VPN, firewall, reverse proxy или другой внешний контур доступа.
|
||||
|
||||
## Обслуживание
|
||||
|
||||
```bash
|
||||
docker compose logs -f api
|
||||
docker compose logs -f worker
|
||||
docker compose exec postgres pg_dump -U miem miem_workers > backup.sql
|
||||
docker compose down
|
||||
```
|
||||
|
||||
Версия сервиса: `0.7.0`. Админка всегда показывает версии backend и frontend в footer.
|
||||
- неизмененные профили учитываются в `crawl_runs.skipped_count` и не получают новый snapshot.
|
||||
|
||||
Во время выполнения парсинга `found_count`, `parsed_count`, `skipped_count` и `error_count` обновляются в базе. Админка опрашивает `/api/crawl-runs/latest` и показывает прогресс как `(parsed_count + skipped_count + error_count) / found_count`.
|
||||
|
||||
## MCP
|
||||
|
||||
Endpoint: `POST /mcp`, без авторизации на уровне приложения.
|
||||
|
||||
Поддерживаемые tools:
|
||||
|
||||
- `get_service_info()`
|
||||
- `sync_employees(client_hash?, include_data?)`
|
||||
- `search_employees(query, status?, limit?)`
|
||||
- `get_employee(profile_id_or_url)`
|
||||
- `list_employee_publications(profile_id_or_url)` — публикации сотрудника; при наличии данных из `employee_publications` возвращает авторов, DOI, аннотацию, описание, citation text, год, тип, язык, статус и ссылку HSE Publications.
|
||||
- `list_employee_courses(profile_id_or_url)`
|
||||
- `get_crawl_status()`
|
||||
- `get_crawl_run_details(run_id)`
|
||||
|
||||
`get_service_info` возвращает метаданные сервиса, список tools и текущую версию набора сотрудников. `sync_employees` отдает полный snapshot или delta по `client_hash`; checksum набора строится по сотрудникам, их статусам и текущим checksums. Ответы tools возвращаются как JSON-строка внутри MCP `content[0].text`.
|
||||
|
||||
Новости сотрудника отдельной MCP tool не имеют: они доступны в `get_employee(...).data.sections` и `sync_employees(include_data=true)` как секция `type = "news"` с массивом `news_links`.
|
||||
|
||||
Пример локального запроса списка tools:
|
||||
|
||||
```bash
|
||||
curl http://localhost:8001/mcp \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}'
|
||||
```
|
||||
|
||||
Если MCP нужно ограничить, делайте это на сетевом уровне: localhost binding, VPN, firewall, reverse proxy или другой внешний контур доступа.
|
||||
|
||||
## Обслуживание
|
||||
|
||||
```bash
|
||||
docker compose logs -f api
|
||||
docker compose logs -f worker
|
||||
docker compose exec postgres pg_dump -U miem miem_workers > backup.sql
|
||||
docker compose down
|
||||
```
|
||||
|
||||
Версия сервиса: `0.7.1`. Админка всегда показывает версии backend и frontend в footer.
|
||||
|
||||
489
app/admin.py
489
app/admin.py
@@ -1,219 +1,238 @@
|
||||
from fastapi import APIRouter, BackgroundTasks, Depends, Form, Request
|
||||
from fastapi.responses import HTMLResponse, RedirectResponse
|
||||
from fastapi.templating import Jinja2Templates
|
||||
from sqlalchemy import desc, func, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.config import Settings, get_settings
|
||||
from app.db import SessionLocal, get_db
|
||||
from app.models import CrawlError, CrawlRun, Employee
|
||||
from app.security import SESSION_COOKIE, require_admin, sign_session, verify_admin
|
||||
from app.services.admin_data import (
|
||||
employee_detail_payload,
|
||||
format_admin_datetime,
|
||||
list_employees_page,
|
||||
run_detail_payload,
|
||||
run_payload,
|
||||
stats_payload,
|
||||
)
|
||||
from app.services.crawl_control import get_running_run, run_crawl_if_idle
|
||||
from app.services.crawler import refresh_employee
|
||||
from app.version import BACKEND_VERSION, FRONTEND_VERSION
|
||||
|
||||
router = APIRouter(prefix="/admin")
|
||||
templates = Jinja2Templates(directory="app/templates")
|
||||
|
||||
|
||||
@router.get("", response_class=HTMLResponse)
|
||||
def dashboard(request: Request, db: Session = Depends(get_db), settings: Settings = Depends(get_settings)):
|
||||
require_admin(request, settings)
|
||||
counts = stats_payload(db)
|
||||
counts["runs"] = db.scalar(select(func.count()).select_from(CrawlRun)) or 0
|
||||
counts["errors"] = db.scalar(select(func.count()).select_from(CrawlError)) or 0
|
||||
run_models = db.scalars(select(CrawlRun).order_by(desc(CrawlRun.started_at)).limit(5)).all()
|
||||
runs = [run_payload(run) for run in run_models]
|
||||
return _render(request, "dashboard.html", {"counts": counts, "runs": runs, "latest_run": runs[0] if runs else None})
|
||||
|
||||
|
||||
@router.get("/login", response_class=HTMLResponse)
|
||||
def login_form(request: Request):
|
||||
return _render(request, "login.html", {"error": None})
|
||||
|
||||
|
||||
@router.post("/login")
|
||||
def login(
|
||||
request: Request,
|
||||
username: str = Form(...),
|
||||
password: str = Form(...),
|
||||
settings: Settings = Depends(get_settings),
|
||||
):
|
||||
if not verify_admin(username, password, settings):
|
||||
return _render(request, "login.html", {"error": "Неверный логин или пароль"}, status_code=401)
|
||||
redirect = RedirectResponse("/admin", status_code=303)
|
||||
redirect.set_cookie(SESSION_COOKIE, sign_session(username, settings), httponly=True, samesite="lax")
|
||||
return redirect
|
||||
|
||||
|
||||
@router.post("/logout")
|
||||
def logout():
|
||||
redirect = RedirectResponse("/admin/login", status_code=303)
|
||||
redirect.delete_cookie(SESSION_COOKIE)
|
||||
return redirect
|
||||
|
||||
|
||||
@router.get("/employees", response_class=HTMLResponse)
|
||||
def employees(
|
||||
request: Request,
|
||||
status: str | None = None,
|
||||
q: str | None = None,
|
||||
settings: Settings = Depends(get_settings),
|
||||
):
|
||||
require_admin(request, settings)
|
||||
return RedirectResponse("/admin/directory", status_code=303)
|
||||
|
||||
|
||||
@router.get("/directory", response_class=HTMLResponse)
|
||||
def directory(
|
||||
request: Request,
|
||||
status: str | None = None,
|
||||
q: str | None = None,
|
||||
started_from: str | None = None,
|
||||
started_to: str | None = None,
|
||||
has_email: str | None = None,
|
||||
sort: str = "full_name",
|
||||
direction: str = "asc",
|
||||
limit: int = 50,
|
||||
offset: int = 0,
|
||||
db: Session = Depends(get_db),
|
||||
settings: Settings = Depends(get_settings),
|
||||
):
|
||||
require_admin(request, settings)
|
||||
parsed_started_from = _parse_date(started_from)
|
||||
parsed_started_to = _parse_date(started_to)
|
||||
parsed_has_email = None if has_email in (None, "") else has_email == "true"
|
||||
page = list_employees_page(
|
||||
db,
|
||||
status=status,
|
||||
q=q,
|
||||
started_from=parsed_started_from,
|
||||
started_to=parsed_started_to,
|
||||
has_email=parsed_has_email,
|
||||
sort=sort,
|
||||
direction=direction,
|
||||
limit=limit,
|
||||
offset=offset,
|
||||
)
|
||||
return _render(
|
||||
request,
|
||||
"directory.html",
|
||||
{
|
||||
"page": page,
|
||||
"filters": {
|
||||
"status": status or "",
|
||||
"q": q or "",
|
||||
"started_from": started_from or "",
|
||||
"started_to": started_to or "",
|
||||
"has_email": has_email or "",
|
||||
"sort": sort,
|
||||
"direction": direction,
|
||||
"limit": page["limit"],
|
||||
"offset": offset,
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@router.get("/employees/{employee_id}", response_class=HTMLResponse)
|
||||
def employee_detail(
|
||||
employee_id: int,
|
||||
request: Request,
|
||||
db: Session = Depends(get_db),
|
||||
settings: Settings = Depends(get_settings),
|
||||
):
|
||||
require_admin(request, settings)
|
||||
employee = db.get(Employee, employee_id)
|
||||
if not employee:
|
||||
return RedirectResponse("/admin/employees", status_code=303)
|
||||
snapshots = [
|
||||
{
|
||||
"captured_display": format_admin_datetime(snapshot.captured_at),
|
||||
"checksum": snapshot.checksum,
|
||||
"parser_version": snapshot.parser_version,
|
||||
}
|
||||
for snapshot in sorted(employee.snapshots, key=lambda item: item.captured_at, reverse=True)[:20]
|
||||
]
|
||||
return _render(
|
||||
request,
|
||||
"employee_detail.html",
|
||||
{
|
||||
"employee": employee,
|
||||
"employee_view": employee_detail_payload(employee),
|
||||
"snapshots": snapshots,
|
||||
"refresh_status": request.query_params.get("refresh_status"),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@router.post("/employees/{employee_id}/refresh")
|
||||
def refresh_employee_detail(
|
||||
employee_id: int,
|
||||
request: Request,
|
||||
db: Session = Depends(get_db),
|
||||
settings: Settings = Depends(get_settings),
|
||||
):
|
||||
require_admin(request, settings)
|
||||
employee = db.get(Employee, employee_id)
|
||||
if not employee:
|
||||
return RedirectResponse("/admin/directory", status_code=303)
|
||||
run = refresh_employee(db, employee, settings)
|
||||
status = "success" if run.status == "completed" else "error"
|
||||
return RedirectResponse(f"/admin/employees/{employee_id}?refresh_status={status}", status_code=303)
|
||||
|
||||
|
||||
@router.get("/runs", response_class=HTMLResponse)
|
||||
def runs(request: Request, db: Session = Depends(get_db), settings: Settings = Depends(get_settings)):
|
||||
require_admin(request, settings)
|
||||
run_models = db.scalars(select(CrawlRun).order_by(desc(CrawlRun.started_at)).limit(50)).all()
|
||||
items = [run_payload(run) for run in run_models]
|
||||
errors = db.scalars(select(CrawlError).order_by(desc(CrawlError.created_at)).limit(50)).all()
|
||||
return _render(request, "runs.html", {"runs": items, "errors": errors})
|
||||
|
||||
|
||||
@router.get("/runs/{run_id}", response_class=HTMLResponse)
|
||||
def run_detail(
|
||||
run_id: int,
|
||||
request: Request,
|
||||
db: Session = Depends(get_db),
|
||||
settings: Settings = Depends(get_settings),
|
||||
):
|
||||
require_admin(request, settings)
|
||||
run = db.get(CrawlRun, run_id)
|
||||
if not run:
|
||||
return RedirectResponse("/admin/runs", status_code=303)
|
||||
return _render(request, "run_detail.html", {"run": run_detail_payload(db, run)})
|
||||
|
||||
|
||||
@router.post("/runs")
|
||||
def trigger_run(
|
||||
request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
db: Session = Depends(get_db),
|
||||
settings: Settings = Depends(get_settings),
|
||||
):
|
||||
require_admin(request, settings)
|
||||
if get_running_run(db):
|
||||
return RedirectResponse("/admin/runs", status_code=303)
|
||||
|
||||
def _crawl() -> None:
|
||||
with SessionLocal() as db:
|
||||
run_crawl_if_idle(db, settings)
|
||||
|
||||
background_tasks.add_task(_crawl)
|
||||
return RedirectResponse("/admin/runs", status_code=303)
|
||||
|
||||
|
||||
from fastapi import APIRouter, BackgroundTasks, Depends, Form, Request
|
||||
from fastapi.responses import HTMLResponse, RedirectResponse
|
||||
from fastapi.templating import Jinja2Templates
|
||||
from sqlalchemy import desc, func, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.config import Settings, get_settings
|
||||
from app.db import SessionLocal, get_db
|
||||
from app.models import CrawlError, CrawlRun, Employee
|
||||
from app.security import SESSION_COOKIE, require_admin, sign_session, verify_admin
|
||||
from app.services.admin_data import (
|
||||
employee_detail_payload,
|
||||
format_admin_datetime,
|
||||
list_employees_page,
|
||||
run_detail_payload,
|
||||
run_payload,
|
||||
stats_payload,
|
||||
)
|
||||
from app.services.crawl_control import get_running_run, run_crawl_if_idle
|
||||
from app.services.crawler import refresh_dismissed_status, refresh_employee
|
||||
from app.version import BACKEND_VERSION, FRONTEND_VERSION
|
||||
|
||||
router = APIRouter(prefix="/admin")
|
||||
templates = Jinja2Templates(directory="app/templates")
|
||||
|
||||
|
||||
@router.get("", response_class=HTMLResponse)
|
||||
def dashboard(request: Request, db: Session = Depends(get_db), settings: Settings = Depends(get_settings)):
|
||||
require_admin(request, settings)
|
||||
counts = stats_payload(db)
|
||||
counts["runs"] = db.scalar(select(func.count()).select_from(CrawlRun)) or 0
|
||||
counts["errors"] = db.scalar(select(func.count()).select_from(CrawlError)) or 0
|
||||
run_models = db.scalars(select(CrawlRun).order_by(desc(CrawlRun.started_at)).limit(5)).all()
|
||||
runs = [run_payload(run) for run in run_models]
|
||||
return _render(request, "dashboard.html", {"counts": counts, "runs": runs, "latest_run": runs[0] if runs else None})
|
||||
|
||||
|
||||
@router.get("/login", response_class=HTMLResponse)
|
||||
def login_form(request: Request):
|
||||
return _render(request, "login.html", {"error": None})
|
||||
|
||||
|
||||
@router.post("/login")
|
||||
def login(
|
||||
request: Request,
|
||||
username: str = Form(...),
|
||||
password: str = Form(...),
|
||||
settings: Settings = Depends(get_settings),
|
||||
):
|
||||
if not verify_admin(username, password, settings):
|
||||
return _render(request, "login.html", {"error": "Неверный логин или пароль"}, status_code=401)
|
||||
redirect = RedirectResponse("/admin", status_code=303)
|
||||
redirect.set_cookie(SESSION_COOKIE, sign_session(username, settings), httponly=True, samesite="lax")
|
||||
return redirect
|
||||
|
||||
|
||||
@router.post("/logout")
|
||||
def logout():
|
||||
redirect = RedirectResponse("/admin/login", status_code=303)
|
||||
redirect.delete_cookie(SESSION_COOKIE)
|
||||
return redirect
|
||||
|
||||
|
||||
@router.get("/employees", response_class=HTMLResponse)
|
||||
def employees(
|
||||
request: Request,
|
||||
status: str | None = None,
|
||||
q: str | None = None,
|
||||
settings: Settings = Depends(get_settings),
|
||||
):
|
||||
require_admin(request, settings)
|
||||
return RedirectResponse("/admin/directory", status_code=303)
|
||||
|
||||
|
||||
@router.get("/directory", response_class=HTMLResponse)
|
||||
def directory(
|
||||
request: Request,
|
||||
status: str | None = None,
|
||||
q: str | None = None,
|
||||
started_from: str | None = None,
|
||||
started_to: str | None = None,
|
||||
has_email: str | None = None,
|
||||
sort: str = "full_name",
|
||||
direction: str = "asc",
|
||||
limit: int = 50,
|
||||
offset: int = 0,
|
||||
db: Session = Depends(get_db),
|
||||
settings: Settings = Depends(get_settings),
|
||||
):
|
||||
require_admin(request, settings)
|
||||
parsed_started_from = _parse_date(started_from)
|
||||
parsed_started_to = _parse_date(started_to)
|
||||
parsed_has_email = None if has_email in (None, "") else has_email == "true"
|
||||
page = list_employees_page(
|
||||
db,
|
||||
status=status,
|
||||
q=q,
|
||||
started_from=parsed_started_from,
|
||||
started_to=parsed_started_to,
|
||||
has_email=parsed_has_email,
|
||||
sort=sort,
|
||||
direction=direction,
|
||||
limit=limit,
|
||||
offset=offset,
|
||||
)
|
||||
return _render(
|
||||
request,
|
||||
"directory.html",
|
||||
{
|
||||
"page": page,
|
||||
"filters": {
|
||||
"status": status or "",
|
||||
"q": q or "",
|
||||
"started_from": started_from or "",
|
||||
"started_to": started_to or "",
|
||||
"has_email": has_email or "",
|
||||
"sort": sort,
|
||||
"direction": direction,
|
||||
"limit": page["limit"],
|
||||
"offset": offset,
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@router.get("/employees/{employee_id}", response_class=HTMLResponse)
|
||||
def employee_detail(
|
||||
employee_id: int,
|
||||
request: Request,
|
||||
db: Session = Depends(get_db),
|
||||
settings: Settings = Depends(get_settings),
|
||||
):
|
||||
require_admin(request, settings)
|
||||
employee = db.get(Employee, employee_id)
|
||||
if not employee:
|
||||
return RedirectResponse("/admin/employees", status_code=303)
|
||||
snapshots = [
|
||||
{
|
||||
"captured_display": format_admin_datetime(snapshot.captured_at),
|
||||
"checksum": snapshot.checksum,
|
||||
"parser_version": snapshot.parser_version,
|
||||
}
|
||||
for snapshot in sorted(employee.snapshots, key=lambda item: item.captured_at, reverse=True)[:20]
|
||||
]
|
||||
return _render(
|
||||
request,
|
||||
"employee_detail.html",
|
||||
{
|
||||
"employee": employee,
|
||||
"employee_view": employee_detail_payload(employee),
|
||||
"snapshots": snapshots,
|
||||
"refresh_status": request.query_params.get("refresh_status"),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@router.post("/employees/{employee_id}/refresh")
|
||||
def refresh_employee_detail(
|
||||
employee_id: int,
|
||||
request: Request,
|
||||
db: Session = Depends(get_db),
|
||||
settings: Settings = Depends(get_settings),
|
||||
):
|
||||
require_admin(request, settings)
|
||||
employee = db.get(Employee, employee_id)
|
||||
if not employee:
|
||||
return RedirectResponse("/admin/directory", status_code=303)
|
||||
run = refresh_employee(db, employee, settings)
|
||||
status = "success" if run.status == "completed" else "error"
|
||||
return RedirectResponse(f"/admin/employees/{employee_id}?refresh_status={status}", status_code=303)
|
||||
|
||||
|
||||
@router.get("/runs", response_class=HTMLResponse)
|
||||
def runs(request: Request, db: Session = Depends(get_db), settings: Settings = Depends(get_settings)):
|
||||
require_admin(request, settings)
|
||||
run_models = db.scalars(select(CrawlRun).order_by(desc(CrawlRun.started_at)).limit(50)).all()
|
||||
items = [run_payload(run) for run in run_models]
|
||||
errors = db.scalars(select(CrawlError).order_by(desc(CrawlError.created_at)).limit(50)).all()
|
||||
return _render(request, "runs.html", {"runs": items, "errors": errors})
|
||||
|
||||
|
||||
@router.get("/runs/{run_id}", response_class=HTMLResponse)
|
||||
def run_detail(
|
||||
run_id: int,
|
||||
request: Request,
|
||||
db: Session = Depends(get_db),
|
||||
settings: Settings = Depends(get_settings),
|
||||
):
|
||||
require_admin(request, settings)
|
||||
run = db.get(CrawlRun, run_id)
|
||||
if not run:
|
||||
return RedirectResponse("/admin/runs", status_code=303)
|
||||
return _render(request, "run_detail.html", {"run": run_detail_payload(db, run)})
|
||||
|
||||
|
||||
@router.post("/runs")
|
||||
def trigger_run(
|
||||
request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
db: Session = Depends(get_db),
|
||||
settings: Settings = Depends(get_settings),
|
||||
):
|
||||
require_admin(request, settings)
|
||||
if get_running_run(db):
|
||||
return RedirectResponse("/admin/runs", status_code=303)
|
||||
|
||||
def _crawl() -> None:
|
||||
with SessionLocal() as db:
|
||||
run_crawl_if_idle(db, settings)
|
||||
|
||||
background_tasks.add_task(_crawl)
|
||||
return RedirectResponse("/admin/runs", status_code=303)
|
||||
|
||||
|
||||
@router.post("/crawl-now")
|
||||
def crawl_now(
|
||||
request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
db: Session = Depends(get_db),
|
||||
settings: Settings = Depends(get_settings),
|
||||
):
|
||||
require_admin(request, settings)
|
||||
if get_running_run(db):
|
||||
return RedirectResponse("/admin", status_code=303)
|
||||
|
||||
def _crawl() -> None:
|
||||
with SessionLocal() as db:
|
||||
run_crawl_if_idle(db, settings)
|
||||
|
||||
background_tasks.add_task(_crawl)
|
||||
return RedirectResponse("/admin", status_code=303)
|
||||
|
||||
|
||||
@router.post("/dismissed/refresh")
|
||||
def refresh_dismissed(
|
||||
request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
db: Session = Depends(get_db),
|
||||
@@ -223,30 +242,30 @@ def crawl_now(
|
||||
if get_running_run(db):
|
||||
return RedirectResponse("/admin", status_code=303)
|
||||
|
||||
def _crawl() -> None:
|
||||
def _refresh() -> None:
|
||||
with SessionLocal() as db:
|
||||
run_crawl_if_idle(db, settings)
|
||||
refresh_dismissed_status(db, settings)
|
||||
|
||||
background_tasks.add_task(_crawl)
|
||||
background_tasks.add_task(_refresh)
|
||||
return RedirectResponse("/admin", status_code=303)
|
||||
|
||||
|
||||
def _render(request: Request, template: str, context: dict, status_code: int = 200) -> HTMLResponse:
|
||||
payload = {
|
||||
"request": request,
|
||||
"backend_version": BACKEND_VERSION,
|
||||
"frontend_version": FRONTEND_VERSION,
|
||||
**context,
|
||||
}
|
||||
return templates.TemplateResponse(request, template, payload, status_code=status_code)
|
||||
|
||||
|
||||
def _parse_date(value: str | None):
|
||||
if not value:
|
||||
return None
|
||||
try:
|
||||
from datetime import date
|
||||
|
||||
return date.fromisoformat(value)
|
||||
except ValueError:
|
||||
return None
|
||||
payload = {
|
||||
"request": request,
|
||||
"backend_version": BACKEND_VERSION,
|
||||
"frontend_version": FRONTEND_VERSION,
|
||||
**context,
|
||||
}
|
||||
return templates.TemplateResponse(request, template, payload, status_code=status_code)
|
||||
|
||||
|
||||
def _parse_date(value: str | None):
|
||||
if not value:
|
||||
return None
|
||||
try:
|
||||
from datetime import date
|
||||
|
||||
return date.fromisoformat(value)
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
1286
app/static/admin.css
1286
app/static/admin.css
File diff suppressed because it is too large
Load Diff
@@ -1,63 +1,69 @@
|
||||
{% extends "base.html" %}
|
||||
{% block title %}Обзор · MIEM Employees{% endblock %}
|
||||
{% block content %}
|
||||
<section class="admin__grid">
|
||||
<a class="metric metric--link" href="/admin/directory"><span class="metric__label">Всего в базе</span><span class="metric__value">{{ counts.total }}</span></a>
|
||||
<a class="metric metric--link" href="/admin/directory?status=active"><span class="metric__label">Работают</span><span class="metric__value">{{ counts.active }}</span></a>
|
||||
<a class="metric metric--link" href="{% if latest_run %}/admin/runs/{{ latest_run.id }}#new-employees{% else %}/admin/runs{% endif %}"><span class="metric__label">Новые за запуск</span><span class="metric__value">{{ counts.new_in_last_run }}</span></a>
|
||||
<a class="metric metric--link" href="/admin/directory?status=dismissed"><span class="metric__label">Уволены</span><span class="metric__value">{{ counts.dismissed }}</span></a>
|
||||
</section>
|
||||
<section class="stats-strip">
|
||||
<div class="stats-strip__item">
|
||||
<span class="stats-strip__label">Последний добавленный</span>
|
||||
{% if counts.latest_added %}
|
||||
<a class="stats-strip__value" href="/admin/employees/{{ counts.latest_added.id }}">{{ counts.latest_added.full_name or counts.latest_added.canonical_url }}</a>
|
||||
{% else %}
|
||||
<span class="stats-strip__value">Сотрудников пока нет</span>
|
||||
{% endif %}
|
||||
</div>
|
||||
<a class="stats-strip__item stats-strip__item--link" href="/admin/runs">
|
||||
<span class="stats-strip__label">Запуски</span>
|
||||
<span class="stats-strip__value">{{ counts.runs }}</span>
|
||||
</a>
|
||||
<div class="stats-strip__item">
|
||||
<span class="stats-strip__label">Ошибки</span>
|
||||
<span class="stats-strip__value">{{ counts.errors }}</span>
|
||||
</div>
|
||||
</section>
|
||||
<section class="panel progress-panel" data-progress-panel>
|
||||
<div class="progress-panel__header">
|
||||
{% extends "base.html" %}
|
||||
{% block title %}Обзор · MIEM Employees{% endblock %}
|
||||
{% block content %}
|
||||
<section class="admin__grid">
|
||||
<a class="metric metric--link" href="/admin/directory"><span class="metric__label">Всего в базе</span><span class="metric__value">{{ counts.total }}</span></a>
|
||||
<a class="metric metric--link" href="/admin/directory?status=active"><span class="metric__label">Работают</span><span class="metric__value">{{ counts.active }}</span></a>
|
||||
<a class="metric metric--link" href="/admin/directory?status=verification_required"><span class="metric__label">Требуют проверки</span><span class="metric__value">{{ counts.verification_required }}</span></a>
|
||||
<a class="metric metric--link" href="{% if latest_run %}/admin/runs/{{ latest_run.id }}#new-employees{% else %}/admin/runs{% endif %}"><span class="metric__label">Новые за запуск</span><span class="metric__value">{{ counts.new_in_last_run }}</span></a>
|
||||
<a class="metric metric--link" href="/admin/directory?status=dismissed"><span class="metric__label">Уволены</span><span class="metric__value">{{ counts.dismissed }}</span></a>
|
||||
</section>
|
||||
<section class="stats-strip">
|
||||
<div class="stats-strip__item">
|
||||
<span class="stats-strip__label">Последний добавленный</span>
|
||||
{% if counts.latest_added %}
|
||||
<a class="stats-strip__value" href="/admin/employees/{{ counts.latest_added.id }}">{{ counts.latest_added.full_name or counts.latest_added.canonical_url }}</a>
|
||||
{% else %}
|
||||
<span class="stats-strip__value">Сотрудников пока нет</span>
|
||||
{% endif %}
|
||||
</div>
|
||||
<a class="stats-strip__item stats-strip__item--link" href="/admin/runs">
|
||||
<span class="stats-strip__label">Запуски</span>
|
||||
<span class="stats-strip__value">{{ counts.runs }}</span>
|
||||
</a>
|
||||
<div class="stats-strip__item">
|
||||
<span class="stats-strip__label">Ошибки</span>
|
||||
<span class="stats-strip__value">{{ counts.errors }}</span>
|
||||
</div>
|
||||
</section>
|
||||
<section class="panel progress-panel" data-progress-panel>
|
||||
<div class="progress-panel__header">
|
||||
<h2 class="panel__title">Прогресс парсинга</h2>
|
||||
<form method="post" action="/admin/crawl-now">
|
||||
<button class="button" type="submit">Запустить парсинг</button>
|
||||
</form>
|
||||
</div>
|
||||
{% set run = counts.current_running_run or latest_run %}
|
||||
<div class="progress-panel__body" data-progress-body>
|
||||
<div class="progress-panel__meta">
|
||||
<span data-progress-status>{{ run.status_display if run else "Ожидание" }}</span>
|
||||
<span>обработано: <span data-progress-processed>{{ run.processed_count if run else 0 }}</span> / <span data-progress-found>{{ run.found_count if run else 0 }}</span></span>
|
||||
<span>без изменений: <span data-progress-skipped>{{ run.skipped_count if run else 0 }}</span></span>
|
||||
<span>ошибок: <span data-progress-errors>{{ run.error_count if run else 0 }}</span></span>
|
||||
<div class="progress-panel__actions">
|
||||
<form method="post" action="/admin/crawl-now">
|
||||
<button class="button" type="submit">Запустить парсинг</button>
|
||||
</form>
|
||||
<form method="post" action="/admin/dismissed/refresh">
|
||||
<button class="button button--secondary" type="submit">Проверить уволенных</button>
|
||||
</form>
|
||||
</div>
|
||||
<div class="progress-bar" aria-label="Parsing progress">
|
||||
<div class="progress-bar__fill" data-progress-fill style="width: {{ run.progress_percent if run else 0 }}%"></div>
|
||||
</div>
|
||||
<div class="progress-panel__percent"><span data-progress-percent>{{ run.progress_percent if run else 0 }}</span>%</div>
|
||||
</div>
|
||||
</section>
|
||||
<section class="panel">
|
||||
<h2 class="panel__title">Последние запуски</h2>
|
||||
<table class="table">
|
||||
<thead><tr><th class="table__head">ID</th><th class="table__head">Статус</th><th class="table__head">Обработано</th><th class="table__head">Без изменений</th><th class="table__head">Ошибки</th><th class="table__head">Старт</th></tr></thead>
|
||||
<tbody>
|
||||
{% for run in runs %}
|
||||
<tr class="table__row" onclick="window.location.href='/admin/runs/{{ run.id }}'" onkeydown="if (event.key === 'Enter' || event.key === ' ') { event.preventDefault(); window.location.href='/admin/runs/{{ run.id }}'; }" role="link" tabindex="0"><td class="table__cell">{{ run.id }}</td><td class="table__cell">{{ run.status_display }}</td><td class="table__cell">{{ run.parsed_count }}</td><td class="table__cell">{{ run.skipped_count }}</td><td class="table__cell">{{ run.error_count }}</td><td class="table__cell">{{ run.started_display }}</td></tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
</section>
|
||||
{% endblock %}
|
||||
{% block scripts %}
|
||||
<script src="/static/admin.js"></script>
|
||||
{% endblock %}
|
||||
</div>
|
||||
{% set run = counts.current_running_run or latest_run %}
|
||||
<div class="progress-panel__body" data-progress-body>
|
||||
<div class="progress-panel__meta">
|
||||
<span data-progress-status>{{ run.status_display if run else "Ожидание" }}</span>
|
||||
<span>обработано: <span data-progress-processed>{{ run.processed_count if run else 0 }}</span> / <span data-progress-found>{{ run.found_count if run else 0 }}</span></span>
|
||||
<span>без изменений: <span data-progress-skipped>{{ run.skipped_count if run else 0 }}</span></span>
|
||||
<span>ошибок: <span data-progress-errors>{{ run.error_count if run else 0 }}</span></span>
|
||||
</div>
|
||||
<div class="progress-bar" aria-label="Parsing progress">
|
||||
<div class="progress-bar__fill" data-progress-fill style="width: {{ run.progress_percent if run else 0 }}%"></div>
|
||||
</div>
|
||||
<div class="progress-panel__percent"><span data-progress-percent>{{ run.progress_percent if run else 0 }}</span>%</div>
|
||||
</div>
|
||||
</section>
|
||||
<section class="panel">
|
||||
<h2 class="panel__title">Последние запуски</h2>
|
||||
<table class="table">
|
||||
<thead><tr><th class="table__head">ID</th><th class="table__head">Статус</th><th class="table__head">Обработано</th><th class="table__head">Без изменений</th><th class="table__head">Ошибки</th><th class="table__head">Старт</th></tr></thead>
|
||||
<tbody>
|
||||
{% for run in runs %}
|
||||
<tr class="table__row" onclick="window.location.href='/admin/runs/{{ run.id }}'" onkeydown="if (event.key === 'Enter' || event.key === ' ') { event.preventDefault(); window.location.href='/admin/runs/{{ run.id }}'; }" role="link" tabindex="0"><td class="table__cell">{{ run.id }}</td><td class="table__cell">{{ run.status_display }}</td><td class="table__cell">{{ run.parsed_count }}</td><td class="table__cell">{{ run.skipped_count }}</td><td class="table__cell">{{ run.error_count }}</td><td class="table__cell">{{ run.started_display }}</td></tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
</section>
|
||||
{% endblock %}
|
||||
{% block scripts %}
|
||||
<script src="/static/admin.js"></script>
|
||||
{% endblock %}
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
APP_VERSION = "0.7.0"
|
||||
FRONTEND_VERSION = "0.7.0"
|
||||
BACKEND_VERSION = "0.7.0"
|
||||
APP_VERSION = "0.7.1"
|
||||
FRONTEND_VERSION = "0.7.1"
|
||||
BACKEND_VERSION = "0.7.1"
|
||||
|
||||
@@ -1,28 +1,28 @@
|
||||
[project]
|
||||
name = "miem-workers"
|
||||
version = "0.7.0"
|
||||
description = "MIEM employees parser, admin API, and MCP server"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = [
|
||||
"apscheduler>=3.10.4",
|
||||
"beautifulsoup4>=4.12.3",
|
||||
"fastapi>=0.115.0",
|
||||
"httpx>=0.27.0",
|
||||
"jinja2>=3.1.4",
|
||||
"lxml>=5.2.0",
|
||||
"psycopg[binary]>=3.2.0",
|
||||
"pydantic-settings>=2.4.0",
|
||||
"python-multipart>=0.0.9",
|
||||
"requests>=2.32.0",
|
||||
"sqlalchemy>=2.0.32",
|
||||
"uvicorn[standard]>=0.30.0",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = [
|
||||
"pytest>=8.3.0",
|
||||
]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
pythonpath = ["."]
|
||||
[project]
|
||||
name = "miem-workers"
|
||||
version = "0.7.1"
|
||||
description = "MIEM employees parser, admin API, and MCP server"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = [
|
||||
"apscheduler>=3.10.4",
|
||||
"beautifulsoup4>=4.12.3",
|
||||
"fastapi>=0.115.0",
|
||||
"httpx>=0.27.0",
|
||||
"jinja2>=3.1.4",
|
||||
"lxml>=5.2.0",
|
||||
"psycopg[binary]>=3.2.0",
|
||||
"pydantic-settings>=2.4.0",
|
||||
"python-multipart>=0.0.9",
|
||||
"requests>=2.32.0",
|
||||
"sqlalchemy>=2.0.32",
|
||||
"uvicorn[standard]>=0.30.0",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = [
|
||||
"pytest>=8.3.0",
|
||||
]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
pythonpath = ["."]
|
||||
|
||||
@@ -1,95 +1,105 @@
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def test_base_navigation_is_russian_and_has_no_legacy_employees_link():
|
||||
template = Path("app/templates/base.html").read_text(encoding="utf-8")
|
||||
|
||||
assert "Обзор" in template
|
||||
assert "Сотрудники" in template
|
||||
assert "Запуски" in template
|
||||
assert "Выйти" in template
|
||||
assert '<a class="admin__brand-link" href="/admin">MIEM Employees</a>' in template
|
||||
assert ">Employees<" not in template
|
||||
assert "/admin/employees" not in template
|
||||
|
||||
|
||||
def test_directory_template_is_russian_and_uses_display_dates():
|
||||
template = Path("app/templates/directory.html").read_text(encoding="utf-8")
|
||||
|
||||
assert "Сотрудники" in template
|
||||
assert "Колонки" in template
|
||||
assert "Применить" in template
|
||||
assert "На странице: {{ value }}" in template
|
||||
assert "{% for value in [25, 50, 100] %}" in template
|
||||
assert "Найдено:" in template
|
||||
assert "Новости" in template
|
||||
assert "employee.news_count" in template
|
||||
assert "employee.first_seen_display" in template
|
||||
assert "employee.last_seen_display" in template
|
||||
assert "employee.dismissed_display" in template
|
||||
assert "Directory" not in template
|
||||
assert "employees found" not in template
|
||||
|
||||
|
||||
def test_admin_employees_route_redirects_to_directory():
|
||||
source = Path("app/admin.py").read_text(encoding="utf-8")
|
||||
|
||||
assert 'RedirectResponse("/admin/directory", status_code=303)' in source
|
||||
|
||||
|
||||
def test_dashboard_limits_latest_runs_to_five():
|
||||
source = Path("app/admin.py").read_text(encoding="utf-8")
|
||||
|
||||
assert "order_by(desc(CrawlRun.started_at)).limit(5)" in source
|
||||
assert "order_by(desc(CrawlRun.started_at)).limit(10)" not in source
|
||||
|
||||
|
||||
def test_runs_template_links_to_run_detail():
|
||||
template = Path("app/templates/runs.html").read_text(encoding="utf-8")
|
||||
|
||||
assert 'onclick="window.location.href=\'/admin/runs/{{ run.id }}\'"' in template
|
||||
assert "onkeydown=\"if (event.key === 'Enter' || event.key === ' ')" in template
|
||||
assert 'role="link"' in template
|
||||
assert 'tabindex="0"' in template
|
||||
assert 'data-row-href="/admin/runs/{{ run.id }}"' not in template
|
||||
assert '<a class="admin__link" href="/admin/runs/{{ run.id }}">' not in template
|
||||
|
||||
|
||||
def test_run_detail_template_extends_base_and_shows_change_groups():
|
||||
template = Path("app/templates/run_detail.html").read_text(encoding="utf-8")
|
||||
|
||||
assert '{% extends "base.html" %}' in template
|
||||
assert 'id="new-employees"' in template
|
||||
assert "Новые сотрудники" in template
|
||||
assert "Потеряшки" in template
|
||||
assert "Уволенные" in template
|
||||
assert "Детализация сотрудников для этого запуска недоступна" in template
|
||||
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def test_base_navigation_is_russian_and_has_no_legacy_employees_link():
|
||||
template = Path("app/templates/base.html").read_text(encoding="utf-8")
|
||||
|
||||
assert "Обзор" in template
|
||||
assert "Сотрудники" in template
|
||||
assert "Запуски" in template
|
||||
assert "Выйти" in template
|
||||
assert '<a class="admin__brand-link" href="/admin">MIEM Employees</a>' in template
|
||||
assert ">Employees<" not in template
|
||||
assert "/admin/employees" not in template
|
||||
|
||||
|
||||
def test_directory_template_is_russian_and_uses_display_dates():
|
||||
template = Path("app/templates/directory.html").read_text(encoding="utf-8")
|
||||
|
||||
assert "Сотрудники" in template
|
||||
assert "Колонки" in template
|
||||
assert "Применить" in template
|
||||
assert "На странице: {{ value }}" in template
|
||||
assert "{% for value in [25, 50, 100] %}" in template
|
||||
assert "Найдено:" in template
|
||||
assert "Новости" in template
|
||||
assert "employee.news_count" in template
|
||||
assert "employee.first_seen_display" in template
|
||||
assert "employee.last_seen_display" in template
|
||||
assert "employee.dismissed_display" in template
|
||||
assert "verification_required" in template
|
||||
assert "Directory" not in template
|
||||
assert "employees found" not in template
|
||||
|
||||
|
||||
def test_admin_employees_route_redirects_to_directory():
|
||||
source = Path("app/admin.py").read_text(encoding="utf-8")
|
||||
|
||||
assert 'RedirectResponse("/admin/directory", status_code=303)' in source
|
||||
|
||||
|
||||
def test_dashboard_limits_latest_runs_to_five():
|
||||
source = Path("app/admin.py").read_text(encoding="utf-8")
|
||||
|
||||
assert "order_by(desc(CrawlRun.started_at)).limit(5)" in source
|
||||
assert "order_by(desc(CrawlRun.started_at)).limit(10)" not in source
|
||||
|
||||
|
||||
def test_runs_template_links_to_run_detail():
|
||||
template = Path("app/templates/runs.html").read_text(encoding="utf-8")
|
||||
|
||||
assert 'onclick="window.location.href=\'/admin/runs/{{ run.id }}\'"' in template
|
||||
assert "onkeydown=\"if (event.key === 'Enter' || event.key === ' ')" in template
|
||||
assert 'role="link"' in template
|
||||
assert 'tabindex="0"' in template
|
||||
assert 'data-row-href="/admin/runs/{{ run.id }}"' not in template
|
||||
assert '<a class="admin__link" href="/admin/runs/{{ run.id }}">' not in template
|
||||
|
||||
|
||||
def test_run_detail_template_extends_base_and_shows_change_groups():
|
||||
template = Path("app/templates/run_detail.html").read_text(encoding="utf-8")
|
||||
|
||||
assert '{% extends "base.html" %}' in template
|
||||
assert 'id="new-employees"' in template
|
||||
assert "Новые сотрудники" in template
|
||||
assert "Потеряшки" in template
|
||||
assert "Требуют проверки" in template
|
||||
assert "Уволенные" in template
|
||||
assert "Детализация сотрудников для этого запуска недоступна" in template
|
||||
|
||||
|
||||
def test_dashboard_metric_cards_link_to_admin_targets():
|
||||
template = Path("app/templates/dashboard.html").read_text(encoding="utf-8")
|
||||
|
||||
assert 'href="/admin/directory"' in template
|
||||
assert 'href="/admin/directory?status=active"' in template
|
||||
assert '/admin/runs/{{ latest_run.id }}#new-employees' in template
|
||||
assert 'href="/admin/directory?status=dismissed"' in template
|
||||
template = Path("app/templates/dashboard.html").read_text(encoding="utf-8")
|
||||
|
||||
assert 'href="/admin/directory"' in template
|
||||
assert 'href="/admin/directory?status=active"' in template
|
||||
assert 'href="/admin/directory?status=verification_required"' in template
|
||||
assert '/admin/runs/{{ latest_run.id }}#new-employees' in template
|
||||
assert 'href="/admin/directory?status=dismissed"' in template
|
||||
assert 'href="/admin/runs"' in template
|
||||
|
||||
|
||||
def test_dashboard_latest_run_rows_link_to_run_detail():
|
||||
def test_dashboard_has_dismissed_status_refresh_action():
|
||||
template = Path("app/templates/dashboard.html").read_text(encoding="utf-8")
|
||||
|
||||
assert 'onclick="window.location.href=\'/admin/runs/{{ run.id }}\'"' in template
|
||||
assert "onkeydown=\"if (event.key === 'Enter' || event.key === ' ')" in template
|
||||
assert 'role="link"' in template
|
||||
assert 'tabindex="0"' in template
|
||||
assert 'data-row-href="/admin/runs/{{ run.id }}"' not in template
|
||||
assert '<a class="admin__link" href="/admin/runs/{{ run.id }}">' not in template
|
||||
|
||||
|
||||
def test_admin_js_supports_keyboard_activation_for_clickable_rows():
|
||||
source = Path("app/static/admin.js").read_text(encoding="utf-8")
|
||||
|
||||
assert 'addEventListener("keydown"' in source
|
||||
assert '"Enter"' in source
|
||||
assert '" "' in source
|
||||
assert 'action="/admin/dismissed/refresh"' in template
|
||||
assert "Проверить уволенных" in template
|
||||
|
||||
|
||||
def test_dashboard_latest_run_rows_link_to_run_detail():
|
||||
template = Path("app/templates/dashboard.html").read_text(encoding="utf-8")
|
||||
|
||||
assert 'onclick="window.location.href=\'/admin/runs/{{ run.id }}\'"' in template
|
||||
assert "onkeydown=\"if (event.key === 'Enter' || event.key === ' ')" in template
|
||||
assert 'role="link"' in template
|
||||
assert 'tabindex="0"' in template
|
||||
assert 'data-row-href="/admin/runs/{{ run.id }}"' not in template
|
||||
assert '<a class="admin__link" href="/admin/runs/{{ run.id }}">' not in template
|
||||
|
||||
|
||||
def test_admin_js_supports_keyboard_activation_for_clickable_rows():
|
||||
source = Path("app/static/admin.js").read_text(encoding="utf-8")
|
||||
|
||||
assert 'addEventListener("keydown"' in source
|
||||
assert '"Enter"' in source
|
||||
assert '" "' in source
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,383 +1,569 @@
|
||||
import gzip
|
||||
from datetime import datetime, timezone
|
||||
|
||||
import gzip
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from app.models import (
|
||||
CrawlError,
|
||||
CrawlRun,
|
||||
CrawlRunEmployeeChange,
|
||||
Employee,
|
||||
EmployeeNewsLink,
|
||||
EmployeePublication,
|
||||
EmployeeSnapshot,
|
||||
ParseResourceCache,
|
||||
CrawlError,
|
||||
CrawlRun,
|
||||
CrawlRunEmployeeChange,
|
||||
Employee,
|
||||
EmployeeNewsLink,
|
||||
EmployeePublication,
|
||||
EmployeeSnapshot,
|
||||
ParseResourceCache,
|
||||
)
|
||||
from app.services.crawler import _checksum, _mark_dismissed, _upsert_employee
|
||||
from app.services.resource_cache import ResourceCache
|
||||
|
||||
|
||||
class FakeResponse:
|
||||
def __init__(self, status_code):
|
||||
self.status_code = status_code
|
||||
|
||||
|
||||
class FakeSession:
|
||||
def __init__(self, statuses):
|
||||
self.statuses = statuses
|
||||
|
||||
def get(self, url, **_kwargs):
|
||||
return FakeResponse(self.statuses[url])
|
||||
|
||||
|
||||
class ConditionalResponse:
|
||||
def __init__(self, status_code, text="", headers=None):
|
||||
self.status_code = status_code
|
||||
self._text = text
|
||||
self.headers = headers or {}
|
||||
self.text_read = False
|
||||
|
||||
@property
|
||||
def text(self):
|
||||
self.text_read = True
|
||||
return self._text
|
||||
|
||||
def raise_for_status(self):
|
||||
return None
|
||||
|
||||
|
||||
from app.config import Settings
|
||||
from app.services.crawler import _checksum, _mark_dismissed, _upsert_employee, refresh_dismissed_status
|
||||
from app.services.resource_cache import ResourceCache
|
||||
|
||||
|
||||
class FakeResponse:
|
||||
def __init__(self, status_code):
|
||||
self.status_code = status_code
|
||||
|
||||
|
||||
class FakeSession:
|
||||
def __init__(self, statuses):
|
||||
self.statuses = statuses
|
||||
|
||||
def get(self, url, **_kwargs):
|
||||
return FakeResponse(self.statuses[url])
|
||||
|
||||
|
||||
class ConditionalResponse:
|
||||
def __init__(self, status_code, text="", headers=None):
|
||||
self.status_code = status_code
|
||||
self._text = text
|
||||
self.headers = headers or {}
|
||||
self.text_read = False
|
||||
|
||||
@property
|
||||
def text(self):
|
||||
self.text_read = True
|
||||
return self._text
|
||||
|
||||
def raise_for_status(self):
|
||||
return None
|
||||
|
||||
|
||||
class ConditionalSession:
|
||||
def __init__(self):
|
||||
self.requests = []
|
||||
self.not_modified_response = ConditionalResponse(304)
|
||||
|
||||
def get(self, url, **kwargs):
|
||||
self.requests.append((url, kwargs))
|
||||
if kwargs["headers"].get("If-None-Match") == '"cached"':
|
||||
return self.not_modified_response
|
||||
def __init__(self):
|
||||
self.requests = []
|
||||
self.not_modified_response = ConditionalResponse(304)
|
||||
|
||||
def get(self, url, **kwargs):
|
||||
self.requests.append((url, kwargs))
|
||||
if kwargs["headers"].get("If-None-Match") == '"cached"':
|
||||
return self.not_modified_response
|
||||
return ConditionalResponse(200, "fresh", {"ETag": '"fresh"'})
|
||||
|
||||
|
||||
def test_mark_dismissed_records_missing_source_when_profile_is_available(db_session):
|
||||
run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
|
||||
db_session.add(run)
|
||||
db_session.add(
|
||||
Employee(
|
||||
profile_key="staff:kept",
|
||||
canonical_url="https://www.hse.ru/staff/kept",
|
||||
status="active",
|
||||
first_seen_at=datetime.now(timezone.utc),
|
||||
last_seen_at=datetime.now(timezone.utc),
|
||||
)
|
||||
def test_refresh_dismissed_status_reactivates_only_profiles_in_source(monkeypatch, db_session):
|
||||
now = datetime.now(timezone.utc)
|
||||
found = Employee(
|
||||
profile_key="staff:returned",
|
||||
canonical_url="https://www.hse.ru/staff/returned",
|
||||
status="dismissed",
|
||||
dismissed_at=now,
|
||||
first_seen_at=now,
|
||||
last_seen_at=now,
|
||||
)
|
||||
db_session.add(
|
||||
Employee(
|
||||
profile_key="staff:missing",
|
||||
canonical_url="https://www.hse.ru/staff/missing",
|
||||
status="active",
|
||||
first_seen_at=datetime.now(timezone.utc),
|
||||
last_seen_at=datetime.now(timezone.utc),
|
||||
)
|
||||
)
|
||||
db_session.commit()
|
||||
|
||||
dismissed = _mark_dismissed(
|
||||
db_session,
|
||||
run,
|
||||
{"staff:kept"},
|
||||
FakeSession({"https://www.hse.ru/staff/missing": 200}),
|
||||
30,
|
||||
)
|
||||
|
||||
assert dismissed == 0
|
||||
assert db_session.query(Employee).filter_by(profile_key="staff:kept").one().status == "active"
|
||||
missing = db_session.query(Employee).filter_by(profile_key="staff:missing").one()
|
||||
assert missing.status == "active"
|
||||
assert missing.dismissed_at is None
|
||||
change = db_session.query(CrawlRunEmployeeChange).one()
|
||||
assert change.change_type == "missing_from_source"
|
||||
assert change.profile_available is True
|
||||
|
||||
|
||||
def test_mark_dismissed_marks_missing_employee_when_profile_is_unavailable(db_session):
|
||||
run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
|
||||
employee = Employee(
|
||||
still_dismissed = Employee(
|
||||
profile_key="staff:gone",
|
||||
canonical_url="https://www.hse.ru/staff/gone",
|
||||
status="active",
|
||||
first_seen_at=datetime.now(timezone.utc),
|
||||
last_seen_at=datetime.now(timezone.utc),
|
||||
status="dismissed",
|
||||
dismissed_at=now,
|
||||
first_seen_at=now,
|
||||
last_seen_at=now,
|
||||
)
|
||||
db_session.add_all([run, employee])
|
||||
db_session.add_all([found, still_dismissed])
|
||||
db_session.commit()
|
||||
|
||||
dismissed = _mark_dismissed(
|
||||
db_session,
|
||||
run,
|
||||
set(),
|
||||
FakeSession({"https://www.hse.ru/staff/gone": 404}),
|
||||
30,
|
||||
monkeypatch.setattr(
|
||||
"app.services.crawler.collect_profile_links",
|
||||
lambda *_args, **_kwargs: ["https://www.hse.ru/staff/returned"],
|
||||
)
|
||||
|
||||
assert dismissed == 1
|
||||
assert employee.status == "dismissed"
|
||||
assert employee.dismissed_at is not None
|
||||
change = db_session.query(CrawlRunEmployeeChange).one()
|
||||
assert change.change_type == "dismissed"
|
||||
assert change.profile_available is False
|
||||
run = refresh_dismissed_status(db_session, Settings())
|
||||
|
||||
|
||||
def test_upsert_employee_increments_new_count_and_records_change_for_new_employee(db_session):
|
||||
run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
|
||||
db_session.add(run)
|
||||
db_session.commit()
|
||||
|
||||
_upsert_employee(
|
||||
db_session,
|
||||
run,
|
||||
{
|
||||
"source_url": "https://www.hse.ru/staff/newperson",
|
||||
"profile_type": "staff",
|
||||
"profile_id": "newperson",
|
||||
"full_name": "New Person",
|
||||
"tabs": [],
|
||||
"sections": [],
|
||||
"parser_version": "0.2.0",
|
||||
"_html": "<html></html>",
|
||||
},
|
||||
)
|
||||
db_session.commit()
|
||||
|
||||
assert run.new_count == 1
|
||||
change = db_session.query(CrawlRunEmployeeChange).one()
|
||||
assert change.change_type == "new"
|
||||
assert change.full_name == "New Person"
|
||||
|
||||
|
||||
def test_resource_cache_uses_etag_and_reuses_cached_body_on_304(db_session):
|
||||
db_session.add(
|
||||
ParseResourceCache(
|
||||
profile_key="staff:cached",
|
||||
resource_key="main-html",
|
||||
method="GET",
|
||||
url="https://www.hse.ru/staff/cached",
|
||||
request_fingerprint="020d59db7b358d9023d0f185bcbf5a9c085d3cf2bf91d92d48eee9147e8d0f01",
|
||||
etag='"cached"',
|
||||
body_hash="cached-hash",
|
||||
body_snapshot=gzip.compress("cached body".encode("utf-8")),
|
||||
parser_version="0.6.0",
|
||||
)
|
||||
)
|
||||
db_session.commit()
|
||||
session = ConditionalSession()
|
||||
|
||||
result = ResourceCache(db_session).fetch_text(
|
||||
session,
|
||||
profile_key="staff:cached",
|
||||
resource_key="main-html",
|
||||
method="GET",
|
||||
url="https://www.hse.ru/staff/cached",
|
||||
headers={"User-Agent": "test"},
|
||||
timeout=10,
|
||||
)
|
||||
|
||||
assert session.requests[0][1]["headers"]["If-None-Match"] == '"cached"'
|
||||
assert result.text == "cached body"
|
||||
assert result.from_cache is True
|
||||
assert session.not_modified_response.text_read is False
|
||||
|
||||
|
||||
def test_upsert_employee_skips_snapshot_when_checksum_is_unchanged(db_session):
|
||||
first_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
|
||||
second_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
|
||||
db_session.add_all([first_run, second_run])
|
||||
db_session.commit()
|
||||
|
||||
_, first_changed = _upsert_employee(db_session, first_run, _parsed_employee("same"))
|
||||
_, second_changed = _upsert_employee(db_session, second_run, _parsed_employee("same"))
|
||||
db_session.commit()
|
||||
|
||||
assert first_changed is True
|
||||
assert second_changed is False
|
||||
assert db_session.query(EmployeeSnapshot).count() == 1
|
||||
|
||||
|
||||
def test_upsert_employee_saves_publications_and_reuses_existing_rows(db_session):
|
||||
first_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
|
||||
second_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
|
||||
db_session.add_all([first_run, second_run])
|
||||
db_session.commit()
|
||||
|
||||
parsed = _parsed_employee("published")
|
||||
parsed["sections"] = [
|
||||
{
|
||||
"type": "publications",
|
||||
"publications": [
|
||||
{
|
||||
"id": "888959076",
|
||||
"publication_id": "888959076",
|
||||
"title": "Detailed Publication",
|
||||
"year": 2023,
|
||||
"publication_type": "ARTICLE",
|
||||
"language": "ru",
|
||||
"status": 1,
|
||||
"url": "https://publications.hse.ru/view/888959076",
|
||||
"doi_url": "https://doi.org/10.1/test",
|
||||
"citation_text": "Detailed citation",
|
||||
"annotation": {"ru": "Аннотация"},
|
||||
"description": {"main": "Detailed citation"},
|
||||
"authors": [{"id": "1", "title_ru": "Автор"}],
|
||||
"raw_data": {"id": "888959076", "title": "Detailed Publication"},
|
||||
}
|
||||
],
|
||||
}
|
||||
]
|
||||
|
||||
employee, _ = _upsert_employee(db_session, first_run, parsed)
|
||||
db_session.commit()
|
||||
_upsert_employee(db_session, second_run, _parsed_employee_with_publication("published"))
|
||||
db_session.commit()
|
||||
|
||||
publications = db_session.query(EmployeePublication).filter_by(employee_id=employee.id).all()
|
||||
assert len(publications) == 1
|
||||
assert publications[0].doi_url == "https://doi.org/10.1/test"
|
||||
assert publications[0].authors == [{"id": "1", "title_ru": "Автор"}]
|
||||
|
||||
|
||||
def test_upsert_employee_records_publication_errors_without_failing_employee(monkeypatch, db_session):
|
||||
run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
|
||||
db_session.add(run)
|
||||
db_session.commit()
|
||||
|
||||
def broken_sync(*_args, **_kwargs):
|
||||
raise RuntimeError("boom")
|
||||
|
||||
monkeypatch.setattr("app.services.crawler._sync_employee_publications", broken_sync)
|
||||
|
||||
employee, changed = _upsert_employee(db_session, run, _parsed_employee_with_publication("error-safe"))
|
||||
db_session.commit()
|
||||
|
||||
assert changed is True
|
||||
assert employee.full_name == "Same Person"
|
||||
assert db_session.query(Employee).filter_by(profile_key="staff:error-safe").one()
|
||||
error = db_session.query(CrawlError).one()
|
||||
assert "публикации" in error.message.lower()
|
||||
|
||||
|
||||
def test_upsert_employee_saves_news_links_and_reuses_existing_rows(db_session):
|
||||
first_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
|
||||
second_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
|
||||
db_session.add_all([first_run, second_run])
|
||||
db_session.commit()
|
||||
|
||||
employee, _ = _upsert_employee(db_session, first_run, _parsed_employee_with_news("news-person"))
|
||||
db_session.commit()
|
||||
_upsert_employee(db_session, second_run, _parsed_employee_with_news("news-person"))
|
||||
db_session.commit()
|
||||
|
||||
news_links = db_session.query(EmployeeNewsLink).filter_by(employee_id=employee.id).all()
|
||||
assert len(news_links) == 1
|
||||
assert news_links[0].title == "News Title"
|
||||
assert news_links[0].url == "https://www.hse.ru/news/1.html"
|
||||
assert news_links[0].published_year == 2026
|
||||
|
||||
|
||||
def test_upsert_employee_records_news_errors_without_failing_employee(monkeypatch, db_session):
|
||||
run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
|
||||
db_session.add(run)
|
||||
db_session.commit()
|
||||
|
||||
def broken_sync(*_args, **_kwargs):
|
||||
raise RuntimeError("boom")
|
||||
|
||||
monkeypatch.setattr("app.services.crawler._sync_employee_news_links", broken_sync)
|
||||
|
||||
employee, changed = _upsert_employee(db_session, run, _parsed_employee_with_news("news-error-safe"))
|
||||
db_session.commit()
|
||||
|
||||
assert changed is True
|
||||
assert employee.full_name == "Same Person"
|
||||
assert db_session.query(Employee).filter_by(profile_key="staff:news-error-safe").one()
|
||||
error = db_session.query(CrawlError).one()
|
||||
assert "новости" in error.message.lower()
|
||||
|
||||
|
||||
def test_checksum_changes_when_widget_data_changes():
|
||||
base = _parsed_employee("widgets")
|
||||
changed = _parsed_employee("widgets")
|
||||
changed["sections"] = [
|
||||
{
|
||||
"type": "publications",
|
||||
"publications": [{"id": "1", "title": "New publication"}],
|
||||
}
|
||||
]
|
||||
|
||||
assert _checksum(base) != _checksum(changed)
|
||||
|
||||
|
||||
def test_checksum_ignores_date_dependent_experience_text():
|
||||
first = _parsed_employee("experience")
|
||||
second = _parsed_employee("experience")
|
||||
first["sections"] = [{"raw_text": "Стаж работы в НИУ ВШЭ: 5 лет"}]
|
||||
second["sections"] = [{"raw_text": "Стаж работы в НИУ ВШЭ: 6 лет"}]
|
||||
|
||||
assert _checksum(first) == _checksum(second)
|
||||
|
||||
|
||||
def _parsed_employee(profile_id: str) -> dict:
|
||||
return {
|
||||
"source_url": f"https://www.hse.ru/staff/{profile_id}",
|
||||
"profile_type": "staff",
|
||||
"profile_id": profile_id,
|
||||
"full_name": "Same Person",
|
||||
"tabs": [],
|
||||
"sections": [],
|
||||
"parser_version": "0.6.0",
|
||||
"_html": "<html></html>",
|
||||
}
|
||||
|
||||
|
||||
def _parsed_employee_with_publication(profile_id: str) -> dict:
|
||||
parsed = _parsed_employee(profile_id)
|
||||
parsed["sections"] = [
|
||||
{
|
||||
"type": "publications",
|
||||
"publications": [
|
||||
{
|
||||
"id": "888959076",
|
||||
"publication_id": "888959076",
|
||||
"title": "Detailed Publication",
|
||||
"year": 2023,
|
||||
"publication_type": "ARTICLE",
|
||||
"language": "ru",
|
||||
"status": 1,
|
||||
"url": "https://publications.hse.ru/view/888959076",
|
||||
"doi_url": "https://doi.org/10.1/test",
|
||||
"citation_text": "Detailed citation",
|
||||
"annotation": {"ru": "Аннотация"},
|
||||
"description": {"main": "Detailed citation"},
|
||||
"authors": [{"id": "1", "title_ru": "Автор"}],
|
||||
"raw_data": {"id": "888959076", "title": "Detailed Publication"},
|
||||
}
|
||||
],
|
||||
}
|
||||
]
|
||||
return parsed
|
||||
|
||||
|
||||
def _parsed_employee_with_news(profile_id: str) -> dict:
|
||||
parsed = _parsed_employee(profile_id)
|
||||
parsed["sections"] = [
|
||||
{
|
||||
"type": "news",
|
||||
"news_links": [
|
||||
{
|
||||
"title": "News Title",
|
||||
"url": "https://www.hse.ru/news/1.html",
|
||||
"summary": "News summary",
|
||||
"published_at": "2026-04-28T00:00:00+00:00",
|
||||
"published_year": 2026,
|
||||
"raw_data": {"title": "News Title", "url": "https://www.hse.ru/news/1.html"},
|
||||
}
|
||||
],
|
||||
}
|
||||
]
|
||||
return parsed
|
||||
assert run.status == "completed"
|
||||
assert run.parsed_count == 1
|
||||
assert run.skipped_count == 1
|
||||
assert found.status == "active"
|
||||
assert found.dismissed_at is None
|
||||
assert still_dismissed.status == "dismissed"
|
||||
|
||||
|
||||
def test_mark_dismissed_records_missing_source_when_profile_is_available(db_session):
|
||||
run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
|
||||
db_session.add(run)
|
||||
db_session.add(
|
||||
Employee(
|
||||
profile_key="staff:kept",
|
||||
canonical_url="https://www.hse.ru/staff/kept",
|
||||
status="active",
|
||||
first_seen_at=datetime.now(timezone.utc),
|
||||
last_seen_at=datetime.now(timezone.utc),
|
||||
)
|
||||
)
|
||||
db_session.add(
|
||||
Employee(
|
||||
profile_key="staff:missing",
|
||||
canonical_url="https://www.hse.ru/staff/missing",
|
||||
status="active",
|
||||
first_seen_at=datetime.now(timezone.utc),
|
||||
last_seen_at=datetime.now(timezone.utc),
|
||||
)
|
||||
)
|
||||
db_session.commit()
|
||||
|
||||
dismissed = _mark_dismissed(
|
||||
db_session,
|
||||
run,
|
||||
{"staff:kept"},
|
||||
FakeSession({"https://www.hse.ru/staff/missing": 200}),
|
||||
30,
|
||||
)
|
||||
|
||||
assert dismissed == 0
|
||||
assert db_session.query(Employee).filter_by(profile_key="staff:kept").one().status == "active"
|
||||
missing = db_session.query(Employee).filter_by(profile_key="staff:missing").one()
|
||||
assert missing.status == "active"
|
||||
assert missing.dismissed_at is None
|
||||
change = db_session.query(CrawlRunEmployeeChange).one()
|
||||
assert change.change_type == "missing_from_source"
|
||||
assert change.profile_available is True
|
||||
|
||||
|
||||
def test_mark_dismissed_requires_consecutive_unavailable_checks(db_session):
|
||||
run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
|
||||
employee = Employee(
|
||||
profile_key="staff:gone",
|
||||
canonical_url="https://www.hse.ru/staff/gone",
|
||||
status="active",
|
||||
first_seen_at=datetime.now(timezone.utc),
|
||||
last_seen_at=datetime.now(timezone.utc),
|
||||
)
|
||||
db_session.add_all([run, employee])
|
||||
db_session.commit()
|
||||
|
||||
first_check = _mark_dismissed(
|
||||
db_session,
|
||||
run,
|
||||
set(),
|
||||
FakeSession({"https://www.hse.ru/staff/gone": 404}),
|
||||
30,
|
||||
confirmation_runs=2,
|
||||
)
|
||||
|
||||
assert first_check == 0
|
||||
assert employee.status == "verification_required"
|
||||
assert employee.dismissed_at is None
|
||||
assert employee.profile_unavailable_streak == 1
|
||||
assert db_session.query(CrawlRunEmployeeChange).one().change_type == "verification_required"
|
||||
|
||||
second_check = _mark_dismissed(
|
||||
db_session,
|
||||
run,
|
||||
set(),
|
||||
FakeSession({"https://www.hse.ru/staff/gone": 404}),
|
||||
30,
|
||||
confirmation_runs=2,
|
||||
)
|
||||
|
||||
assert second_check == 1
|
||||
assert employee.status == "dismissed"
|
||||
assert employee.dismissed_at is not None
|
||||
assert employee.profile_unavailable_streak == 2
|
||||
change = db_session.query(CrawlRunEmployeeChange).order_by(CrawlRunEmployeeChange.id).all()[-1]
|
||||
assert change.change_type == "dismissed"
|
||||
assert change.profile_available is False
|
||||
|
||||
|
||||
def test_mark_dismissed_does_not_dismiss_on_server_error(db_session):
|
||||
run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
|
||||
employee = Employee(
|
||||
profile_key="staff:temporary-error",
|
||||
canonical_url="https://www.hse.ru/staff/temporary-error",
|
||||
status="active",
|
||||
first_seen_at=datetime.now(timezone.utc),
|
||||
last_seen_at=datetime.now(timezone.utc),
|
||||
)
|
||||
db_session.add_all([run, employee])
|
||||
db_session.commit()
|
||||
|
||||
dismissed = _mark_dismissed(
|
||||
db_session,
|
||||
run,
|
||||
set(),
|
||||
FakeSession({"https://www.hse.ru/staff/temporary-error": 503}),
|
||||
30,
|
||||
confirmation_runs=1,
|
||||
)
|
||||
|
||||
assert dismissed == 0
|
||||
assert employee.status == "active"
|
||||
assert employee.profile_unavailable_streak == 0
|
||||
assert db_session.query(CrawlError).one().error_type == "ProfileAvailabilityCheckError"
|
||||
|
||||
|
||||
def test_available_profile_resets_verification_streak(db_session):
|
||||
run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
|
||||
employee = Employee(
|
||||
profile_key="staff:restored",
|
||||
canonical_url="https://www.hse.ru/staff/restored",
|
||||
status="active",
|
||||
first_seen_at=datetime.now(timezone.utc),
|
||||
last_seen_at=datetime.now(timezone.utc),
|
||||
)
|
||||
db_session.add_all([run, employee])
|
||||
db_session.commit()
|
||||
|
||||
_mark_dismissed(
|
||||
db_session,
|
||||
run,
|
||||
set(),
|
||||
FakeSession({"https://www.hse.ru/staff/restored": 404}),
|
||||
30,
|
||||
confirmation_runs=3,
|
||||
)
|
||||
_mark_dismissed(
|
||||
db_session,
|
||||
run,
|
||||
set(),
|
||||
FakeSession({"https://www.hse.ru/staff/restored": 200}),
|
||||
30,
|
||||
confirmation_runs=3,
|
||||
)
|
||||
|
||||
assert employee.status == "active"
|
||||
assert employee.profile_unavailable_streak == 0
|
||||
|
||||
|
||||
def test_mark_dismissed_blocks_mass_auto_dismissals(db_session):
|
||||
run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
|
||||
employees = [
|
||||
Employee(
|
||||
profile_key=f"staff:gone-{index}",
|
||||
canonical_url=f"https://www.hse.ru/staff/gone-{index}",
|
||||
status="active",
|
||||
first_seen_at=datetime.now(timezone.utc),
|
||||
last_seen_at=datetime.now(timezone.utc),
|
||||
)
|
||||
for index in range(2)
|
||||
]
|
||||
db_session.add_all([run, *employees])
|
||||
db_session.commit()
|
||||
|
||||
dismissed = _mark_dismissed(
|
||||
db_session,
|
||||
run,
|
||||
set(),
|
||||
FakeSession({employee.canonical_url: 404 for employee in employees}),
|
||||
30,
|
||||
confirmation_runs=1,
|
||||
max_auto_dismissals=1,
|
||||
)
|
||||
|
||||
assert dismissed == 0
|
||||
assert {employee.status for employee in employees} == {"verification_required"}
|
||||
assert "приостановлено" in run.message
|
||||
|
||||
|
||||
def test_upsert_employee_increments_new_count_and_records_change_for_new_employee(db_session):
|
||||
run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
|
||||
db_session.add(run)
|
||||
db_session.commit()
|
||||
|
||||
_upsert_employee(
|
||||
db_session,
|
||||
run,
|
||||
{
|
||||
"source_url": "https://www.hse.ru/staff/newperson",
|
||||
"profile_type": "staff",
|
||||
"profile_id": "newperson",
|
||||
"full_name": "New Person",
|
||||
"tabs": [],
|
||||
"sections": [],
|
||||
"parser_version": "0.2.0",
|
||||
"_html": "<html></html>",
|
||||
},
|
||||
)
|
||||
db_session.commit()
|
||||
|
||||
assert run.new_count == 1
|
||||
change = db_session.query(CrawlRunEmployeeChange).one()
|
||||
assert change.change_type == "new"
|
||||
assert change.full_name == "New Person"
|
||||
|
||||
|
||||
def test_upsert_employee_reconciles_profile_moved_to_new_url(db_session):
|
||||
run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
|
||||
employee = Employee(
|
||||
profile_key="staff:abelov",
|
||||
canonical_url="https://www.hse.ru/staff/abelov",
|
||||
full_name="Белов Александр Владимирович",
|
||||
status="active",
|
||||
first_seen_at=datetime.now(timezone.utc),
|
||||
last_seen_at=datetime.now(timezone.utc),
|
||||
)
|
||||
db_session.add_all([run, employee])
|
||||
db_session.commit()
|
||||
employee_id = employee.id
|
||||
|
||||
updated, changed = _upsert_employee(
|
||||
db_session,
|
||||
run,
|
||||
{
|
||||
"source_url": "https://www.hse.ru/org/persons/47634735",
|
||||
"profile_type": "org_person",
|
||||
"profile_id": "47634735",
|
||||
"full_name": "Белов Александр Владимирович",
|
||||
"tabs": [],
|
||||
"sections": [],
|
||||
"parser_version": "0.7.0",
|
||||
"_html": "<html></html>",
|
||||
},
|
||||
)
|
||||
db_session.commit()
|
||||
|
||||
assert changed is True
|
||||
assert updated.id == employee_id
|
||||
assert updated.profile_key == "org_person:47634735"
|
||||
assert updated.canonical_url == "https://www.hse.ru/org/persons/47634735"
|
||||
assert updated.status == "active"
|
||||
assert run.new_count == 0
|
||||
assert db_session.query(Employee).count() == 1
|
||||
assert {item.url for item in updated.profile_urls} == {
|
||||
"https://www.hse.ru/staff/abelov",
|
||||
"https://www.hse.ru/org/persons/47634735",
|
||||
}
|
||||
|
||||
|
||||
def test_resource_cache_uses_etag_and_reuses_cached_body_on_304(db_session):
|
||||
db_session.add(
|
||||
ParseResourceCache(
|
||||
profile_key="staff:cached",
|
||||
resource_key="main-html",
|
||||
method="GET",
|
||||
url="https://www.hse.ru/staff/cached",
|
||||
request_fingerprint="020d59db7b358d9023d0f185bcbf5a9c085d3cf2bf91d92d48eee9147e8d0f01",
|
||||
etag='"cached"',
|
||||
body_hash="cached-hash",
|
||||
body_snapshot=gzip.compress("cached body".encode("utf-8")),
|
||||
parser_version="0.6.0",
|
||||
)
|
||||
)
|
||||
db_session.commit()
|
||||
session = ConditionalSession()
|
||||
|
||||
result = ResourceCache(db_session).fetch_text(
|
||||
session,
|
||||
profile_key="staff:cached",
|
||||
resource_key="main-html",
|
||||
method="GET",
|
||||
url="https://www.hse.ru/staff/cached",
|
||||
headers={"User-Agent": "test"},
|
||||
timeout=10,
|
||||
)
|
||||
|
||||
assert session.requests[0][1]["headers"]["If-None-Match"] == '"cached"'
|
||||
assert result.text == "cached body"
|
||||
assert result.from_cache is True
|
||||
assert session.not_modified_response.text_read is False
|
||||
|
||||
|
||||
def test_upsert_employee_skips_snapshot_when_checksum_is_unchanged(db_session):
|
||||
first_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
|
||||
second_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
|
||||
db_session.add_all([first_run, second_run])
|
||||
db_session.commit()
|
||||
|
||||
_, first_changed = _upsert_employee(db_session, first_run, _parsed_employee("same"))
|
||||
_, second_changed = _upsert_employee(db_session, second_run, _parsed_employee("same"))
|
||||
db_session.commit()
|
||||
|
||||
assert first_changed is True
|
||||
assert second_changed is False
|
||||
assert db_session.query(EmployeeSnapshot).count() == 1
|
||||
|
||||
|
||||
def test_upsert_employee_saves_publications_and_reuses_existing_rows(db_session):
|
||||
first_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
|
||||
second_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
|
||||
db_session.add_all([first_run, second_run])
|
||||
db_session.commit()
|
||||
|
||||
parsed = _parsed_employee("published")
|
||||
parsed["sections"] = [
|
||||
{
|
||||
"type": "publications",
|
||||
"publications": [
|
||||
{
|
||||
"id": "888959076",
|
||||
"publication_id": "888959076",
|
||||
"title": "Detailed Publication",
|
||||
"year": 2023,
|
||||
"publication_type": "ARTICLE",
|
||||
"language": "ru",
|
||||
"status": 1,
|
||||
"url": "https://publications.hse.ru/view/888959076",
|
||||
"doi_url": "https://doi.org/10.1/test",
|
||||
"citation_text": "Detailed citation",
|
||||
"annotation": {"ru": "Аннотация"},
|
||||
"description": {"main": "Detailed citation"},
|
||||
"authors": [{"id": "1", "title_ru": "Автор"}],
|
||||
"raw_data": {"id": "888959076", "title": "Detailed Publication"},
|
||||
}
|
||||
],
|
||||
}
|
||||
]
|
||||
|
||||
employee, _ = _upsert_employee(db_session, first_run, parsed)
|
||||
db_session.commit()
|
||||
_upsert_employee(db_session, second_run, _parsed_employee_with_publication("published"))
|
||||
db_session.commit()
|
||||
|
||||
publications = db_session.query(EmployeePublication).filter_by(employee_id=employee.id).all()
|
||||
assert len(publications) == 1
|
||||
assert publications[0].doi_url == "https://doi.org/10.1/test"
|
||||
assert publications[0].authors == [{"id": "1", "title_ru": "Автор"}]
|
||||
|
||||
|
||||
def test_upsert_employee_records_publication_errors_without_failing_employee(monkeypatch, db_session):
|
||||
run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
|
||||
db_session.add(run)
|
||||
db_session.commit()
|
||||
|
||||
def broken_sync(*_args, **_kwargs):
|
||||
raise RuntimeError("boom")
|
||||
|
||||
monkeypatch.setattr("app.services.crawler._sync_employee_publications", broken_sync)
|
||||
|
||||
employee, changed = _upsert_employee(db_session, run, _parsed_employee_with_publication("error-safe"))
|
||||
db_session.commit()
|
||||
|
||||
assert changed is True
|
||||
assert employee.full_name == "Same Person"
|
||||
assert db_session.query(Employee).filter_by(profile_key="staff:error-safe").one()
|
||||
error = db_session.query(CrawlError).one()
|
||||
assert "публикации" in error.message.lower()
|
||||
|
||||
|
||||
def test_upsert_employee_saves_news_links_and_reuses_existing_rows(db_session):
|
||||
first_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
|
||||
second_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
|
||||
db_session.add_all([first_run, second_run])
|
||||
db_session.commit()
|
||||
|
||||
employee, _ = _upsert_employee(db_session, first_run, _parsed_employee_with_news("news-person"))
|
||||
db_session.commit()
|
||||
_upsert_employee(db_session, second_run, _parsed_employee_with_news("news-person"))
|
||||
db_session.commit()
|
||||
|
||||
news_links = db_session.query(EmployeeNewsLink).filter_by(employee_id=employee.id).all()
|
||||
assert len(news_links) == 1
|
||||
assert news_links[0].title == "News Title"
|
||||
assert news_links[0].url == "https://www.hse.ru/news/1.html"
|
||||
assert news_links[0].published_year == 2026
|
||||
|
||||
|
||||
def test_upsert_employee_records_news_errors_without_failing_employee(monkeypatch, db_session):
|
||||
run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
|
||||
db_session.add(run)
|
||||
db_session.commit()
|
||||
|
||||
def broken_sync(*_args, **_kwargs):
|
||||
raise RuntimeError("boom")
|
||||
|
||||
monkeypatch.setattr("app.services.crawler._sync_employee_news_links", broken_sync)
|
||||
|
||||
employee, changed = _upsert_employee(db_session, run, _parsed_employee_with_news("news-error-safe"))
|
||||
db_session.commit()
|
||||
|
||||
assert changed is True
|
||||
assert employee.full_name == "Same Person"
|
||||
assert db_session.query(Employee).filter_by(profile_key="staff:news-error-safe").one()
|
||||
error = db_session.query(CrawlError).one()
|
||||
assert "новости" in error.message.lower()
|
||||
|
||||
|
||||
def test_checksum_changes_when_widget_data_changes():
|
||||
base = _parsed_employee("widgets")
|
||||
changed = _parsed_employee("widgets")
|
||||
changed["sections"] = [
|
||||
{
|
||||
"type": "publications",
|
||||
"publications": [{"id": "1", "title": "New publication"}],
|
||||
}
|
||||
]
|
||||
|
||||
assert _checksum(base) != _checksum(changed)
|
||||
|
||||
|
||||
def test_checksum_ignores_date_dependent_experience_text():
|
||||
first = _parsed_employee("experience")
|
||||
second = _parsed_employee("experience")
|
||||
first["sections"] = [{"raw_text": "Стаж работы в НИУ ВШЭ: 5 лет"}]
|
||||
second["sections"] = [{"raw_text": "Стаж работы в НИУ ВШЭ: 6 лет"}]
|
||||
|
||||
assert _checksum(first) == _checksum(second)
|
||||
|
||||
|
||||
def _parsed_employee(profile_id: str) -> dict:
|
||||
return {
|
||||
"source_url": f"https://www.hse.ru/staff/{profile_id}",
|
||||
"profile_type": "staff",
|
||||
"profile_id": profile_id,
|
||||
"full_name": "Same Person",
|
||||
"tabs": [],
|
||||
"sections": [],
|
||||
"parser_version": "0.6.0",
|
||||
"_html": "<html></html>",
|
||||
}
|
||||
|
||||
|
||||
def _parsed_employee_with_publication(profile_id: str) -> dict:
|
||||
parsed = _parsed_employee(profile_id)
|
||||
parsed["sections"] = [
|
||||
{
|
||||
"type": "publications",
|
||||
"publications": [
|
||||
{
|
||||
"id": "888959076",
|
||||
"publication_id": "888959076",
|
||||
"title": "Detailed Publication",
|
||||
"year": 2023,
|
||||
"publication_type": "ARTICLE",
|
||||
"language": "ru",
|
||||
"status": 1,
|
||||
"url": "https://publications.hse.ru/view/888959076",
|
||||
"doi_url": "https://doi.org/10.1/test",
|
||||
"citation_text": "Detailed citation",
|
||||
"annotation": {"ru": "Аннотация"},
|
||||
"description": {"main": "Detailed citation"},
|
||||
"authors": [{"id": "1", "title_ru": "Автор"}],
|
||||
"raw_data": {"id": "888959076", "title": "Detailed Publication"},
|
||||
}
|
||||
],
|
||||
}
|
||||
]
|
||||
return parsed
|
||||
|
||||
|
||||
def _parsed_employee_with_news(profile_id: str) -> dict:
|
||||
parsed = _parsed_employee(profile_id)
|
||||
parsed["sections"] = [
|
||||
{
|
||||
"type": "news",
|
||||
"news_links": [
|
||||
{
|
||||
"title": "News Title",
|
||||
"url": "https://www.hse.ru/news/1.html",
|
||||
"summary": "News summary",
|
||||
"published_at": "2026-04-28T00:00:00+00:00",
|
||||
"published_year": 2026,
|
||||
"raw_data": {"title": "News Title", "url": "https://www.hse.ru/news/1.html"},
|
||||
}
|
||||
],
|
||||
}
|
||||
]
|
||||
return parsed
|
||||
|
||||
Reference in New Issue
Block a user