forked from admin/miem_workers
Compare commits
6
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e142138c1b | ||
|
|
c031fb9f4a | ||
|
|
70a4719bab | ||
|
|
3bdb8b1d90 | ||
|
|
f3fb714126 | ||
|
|
3ed1dd8832 |
No files matched your search
@@ -1,5 +1,13 @@
|
|||||||
# Changelog
|
# Changelog
|
||||||
|
|
||||||
|
## 0.7.7
|
||||||
|
|
||||||
|
- Удалена неиспользуемая интеграция обмена данными и связанная документация и тесты.
|
||||||
|
|
||||||
|
## 0.7.6
|
||||||
|
|
||||||
|
- Возвращён еженедельный автоматический запуск обхода сотрудников.
|
||||||
|
|
||||||
## 0.7.5
|
## 0.7.5
|
||||||
|
|
||||||
- Production Compose запускает только API и PostgreSQL.
|
- Production Compose запускает только API и PostgreSQL.
|
||||||
|
|||||||
@@ -1,671 +0,0 @@
|
|||||||
# MCP: описание работы, структуры и тулзов
|
|
||||||
|
|
||||||
Документ описывает MCP endpoint сервиса `miem-employees` по текущей реализации в `app/mcp.py`.
|
|
||||||
|
|
||||||
## Где находится MCP
|
|
||||||
|
|
||||||
- FastAPI router: `app.mcp.router`
|
|
||||||
- Подключение к приложению: `app/main.py`
|
|
||||||
- HTTP endpoint: `POST /mcp`
|
|
||||||
- Локально при обычном запуске API: `http://localhost:8000/mcp`
|
|
||||||
- В Docker Compose endpoint обслуживает `api`: `http://localhost:8000/mcp`
|
|
||||||
- Авторизация на уровне приложения: отсутствует. Заголовок `Authorization` не проверяется и не влияет на ответ.
|
|
||||||
|
|
||||||
Если доступ к MCP нужно ограничить, это должно делаться внешним контуром: bind на localhost, VPN, firewall, reverse proxy или отдельная сетевая политика.
|
|
||||||
|
|
||||||
## Протокол
|
|
||||||
|
|
||||||
Endpoint принимает JSON-RPC 2.0 over HTTP.
|
|
||||||
|
|
||||||
Общий формат запроса:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"id": 1,
|
|
||||||
"method": "tools/list",
|
|
||||||
"params": {}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Общий формат успешного ответа:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"id": 1,
|
|
||||||
"result": {}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Общий формат ошибки:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"id": 1,
|
|
||||||
"error": {
|
|
||||||
"code": -32601,
|
|
||||||
"message": "Method not found"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Поддерживаемая версия MCP-протокола:
|
|
||||||
|
|
||||||
```text
|
|
||||||
2024-11-05
|
|
||||||
```
|
|
||||||
|
|
||||||
Имя сервиса:
|
|
||||||
|
|
||||||
```text
|
|
||||||
miem-employees
|
|
||||||
```
|
|
||||||
|
|
||||||
Версия сервера берется из `app.version.BACKEND_VERSION`.
|
|
||||||
|
|
||||||
## Поддерживаемые JSON-RPC методы
|
|
||||||
|
|
||||||
### initialize
|
|
||||||
|
|
||||||
Возвращает метаданные MCP-сервера и capabilities.
|
|
||||||
|
|
||||||
Запрос:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"id": 1,
|
|
||||||
"method": "initialize",
|
|
||||||
"params": {}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Ответ:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"id": 1,
|
|
||||||
"result": {
|
|
||||||
"protocolVersion": "2024-11-05",
|
|
||||||
"serverInfo": {
|
|
||||||
"name": "miem-employees",
|
|
||||||
"version": "0.7.0"
|
|
||||||
},
|
|
||||||
"capabilities": {
|
|
||||||
"tools": {}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### tools/list
|
|
||||||
|
|
||||||
Возвращает список доступных tools с JSON Schema для аргументов.
|
|
||||||
|
|
||||||
Запрос:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"id": 1,
|
|
||||||
"method": "tools/list",
|
|
||||||
"params": {}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Ответ содержит массив `result.tools`.
|
|
||||||
|
|
||||||
### tools/call
|
|
||||||
|
|
||||||
Вызывает один tool по имени.
|
|
||||||
|
|
||||||
Запрос:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"id": 1,
|
|
||||||
"method": "tools/call",
|
|
||||||
"params": {
|
|
||||||
"name": "search_employees",
|
|
||||||
"arguments": {
|
|
||||||
"query": "Сергеев",
|
|
||||||
"limit": 20
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Ответ tool всегда заворачивается в MCP content-массив:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"id": 1,
|
|
||||||
"result": {
|
|
||||||
"content": [
|
|
||||||
{
|
|
||||||
"type": "text",
|
|
||||||
"text": "{\"items\":[]}"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Поле `text` содержит сериализованный JSON с `ensure_ascii=false`. Клиент должен распарсить это поле как JSON, если ему нужна структурированная нагрузка.
|
|
||||||
|
|
||||||
## Ошибки
|
|
||||||
|
|
||||||
- Неизвестный JSON-RPC метод: `code = -32601`, `message = "Method not found"`.
|
|
||||||
- Исключения при обработке tool: `code = -32000`, `message` содержит текст исключения.
|
|
||||||
- Если сущность не найдена внутри отдельных tools, HTTP и JSON-RPC ответ остаются успешными, а полезная нагрузка содержит `{"error": "not_found"}`.
|
|
||||||
|
|
||||||
## Источники данных
|
|
||||||
|
|
||||||
MCP читает данные из основной базы через SQLAlchemy session из `app.db.get_db`.
|
|
||||||
|
|
||||||
Основные таблицы и модели:
|
|
||||||
|
|
||||||
- `employees`: текущая карточка сотрудника, статус, профиль, `current_data`, checksum.
|
|
||||||
- `employee_publications`: нормализованные публикации сотрудников с авторами, DOI, аннотацией, описанием, citation text и raw JSON из HSE Publications.
|
|
||||||
- `employee_news_links`: нормализованные ссылки на новости из блока профиля «В новостях» с заголовком, URL, кратким описанием, датой, годом публикации и raw JSON карточки.
|
|
||||||
- `crawl_runs`: история запусков парсинга.
|
|
||||||
- `crawl_run_employee_changes`: детальные изменения сотрудников в рамках запуска.
|
|
||||||
- `crawl_errors`: ошибки парсинга в рамках запуска.
|
|
||||||
- `dataset_versions`: версии полного набора сотрудников.
|
|
||||||
- `dataset_version_items`: состав конкретной версии набора сотрудников.
|
|
||||||
|
|
||||||
## Общая структура employee payload
|
|
||||||
|
|
||||||
Краткая карточка сотрудника:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"profile_key": "staff:avsergeev",
|
|
||||||
"profile_id": "avsergeev",
|
|
||||||
"full_name": "Сергеев Алексей Викторович",
|
|
||||||
"status": "active",
|
|
||||||
"canonical_url": "https://www.hse.ru/staff/avsergeev",
|
|
||||||
"last_seen_at": "2026-05-14T10:00:00+00:00",
|
|
||||||
"dismissed_at": null
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
В sync payload дополнительно отдается `checksum`.
|
|
||||||
|
|
||||||
Полная карточка дополнительно содержит:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"data": {
|
|
||||||
"contacts": {},
|
|
||||||
"sections": []
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
`data` соответствует распарсенному JSON профиля сотрудника. Внутри `sections` могут быть секции с публикациями, курсами, ВКР, новостями, таблицами, ссылками и произвольными текстовыми блоками.
|
|
||||||
|
|
||||||
Пример секции новостей внутри `data.sections`:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"title": "В новостях",
|
|
||||||
"slug": "v_novostyah",
|
|
||||||
"type": "news",
|
|
||||||
"news_count": 1,
|
|
||||||
"news_links": [
|
|
||||||
{
|
|
||||||
"title": "Название новости",
|
|
||||||
"url": "https://www.hse.ru/news/edu/1153850518.html",
|
|
||||||
"summary": "Краткое описание новости.",
|
|
||||||
"published_at": "2026-04-28T00:00:00+00:00",
|
|
||||||
"published_year": 2026
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Для новостей отдельного MCP tool сейчас нет: они доступны через `get_employee(...).data.sections` или через полную синхронизацию `sync_employees(include_data=true)`.
|
|
||||||
|
|
||||||
## Tools
|
|
||||||
|
|
||||||
### get_service_info
|
|
||||||
|
|
||||||
Назначение: вернуть метаданные сервиса, список tools и текущую версию набора сотрудников.
|
|
||||||
|
|
||||||
Аргументы: отсутствуют.
|
|
||||||
|
|
||||||
Возвращает:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"service_name": "miem-employees",
|
|
||||||
"backend_version": "0.7.0",
|
|
||||||
"protocolVersion": "2024-11-05",
|
|
||||||
"tools": [],
|
|
||||||
"dataset": {
|
|
||||||
"hash": "sha256",
|
|
||||||
"previous_hash": "sha256 или null",
|
|
||||||
"created_at": "2026-05-14T10:00:00+00:00",
|
|
||||||
"crawl_run_id": 123,
|
|
||||||
"employee_count": 100,
|
|
||||||
"active_count": 95,
|
|
||||||
"dismissed_count": 5
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Особенность: перед ответом сервис создает актуальную `dataset_version`, если текущий набор сотрудников еще не имеет версии.
|
|
||||||
|
|
||||||
### sync_employees
|
|
||||||
|
|
||||||
Назначение: синхронизировать клиентский кэш сотрудников по hash набора данных.
|
|
||||||
|
|
||||||
Аргументы:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"client_hash": "sha256 или null",
|
|
||||||
"include_data": true
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
- `client_hash`: hash версии, которая уже есть у клиента. Если не передан, отдается полный snapshot.
|
|
||||||
- `include_data`: управляет включением полного `data` в карточки сотрудников. По умолчанию `true`.
|
|
||||||
|
|
||||||
Полный ответ без `client_hash`:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"mode": "full",
|
|
||||||
"from_hash": null,
|
|
||||||
"to_hash": "current-sha256",
|
|
||||||
"dataset": {},
|
|
||||||
"items": []
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Если клиентский hash совпадает с текущим:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"mode": "delta",
|
|
||||||
"from_hash": "current-sha256",
|
|
||||||
"to_hash": "current-sha256",
|
|
||||||
"dataset": {},
|
|
||||||
"changes": {
|
|
||||||
"added": [],
|
|
||||||
"updated": [],
|
|
||||||
"dismissed": [],
|
|
||||||
"removed": []
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Если `client_hash` неизвестен серверу:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"mode": "full",
|
|
||||||
"from_hash": "missing",
|
|
||||||
"to_hash": "current-sha256",
|
|
||||||
"dataset": {},
|
|
||||||
"items": [],
|
|
||||||
"reason": "unknown_client_hash"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Если `client_hash` найден и отличается от текущего:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"mode": "delta",
|
|
||||||
"from_hash": "old-sha256",
|
|
||||||
"to_hash": "current-sha256",
|
|
||||||
"dataset": {},
|
|
||||||
"changes": {
|
|
||||||
"added": [],
|
|
||||||
"updated": [],
|
|
||||||
"dismissed": [],
|
|
||||||
"removed": []
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Логика delta:
|
|
||||||
|
|
||||||
- `added`: сотрудник появился в новой версии.
|
|
||||||
- `updated`: изменился checksum или статус, и сотрудник активен.
|
|
||||||
- `dismissed`: сотрудник есть в новой версии, но получил статус `dismissed`.
|
|
||||||
- `removed`: `profile_key` был в старой версии, но отсутствует в новой.
|
|
||||||
|
|
||||||
Hash набора считается по отсортированному списку `{profile_key, status, checksum}`.
|
|
||||||
|
|
||||||
### search_employees
|
|
||||||
|
|
||||||
Назначение: найти сотрудников по ФИО или canonical URL.
|
|
||||||
|
|
||||||
Аргументы:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"query": "Сергеев",
|
|
||||||
"status": "active",
|
|
||||||
"limit": 20
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
- `query`: обязательный по schema, но в коде пустая строка означает поиск без текстового фильтра.
|
|
||||||
- `status`: опционально, только `active` или `dismissed`.
|
|
||||||
- `limit`: максимум 100, по умолчанию 20.
|
|
||||||
|
|
||||||
Возвращает массив кратких employee payload без `data`:
|
|
||||||
|
|
||||||
```json
|
|
||||||
[
|
|
||||||
{
|
|
||||||
"profile_key": "staff:avsergeev",
|
|
||||||
"profile_id": "avsergeev",
|
|
||||||
"full_name": "Сергеев Алексей Викторович",
|
|
||||||
"status": "active",
|
|
||||||
"canonical_url": "https://www.hse.ru/staff/avsergeev",
|
|
||||||
"last_seen_at": "2026-05-14T10:00:00+00:00",
|
|
||||||
"dismissed_at": null
|
|
||||||
}
|
|
||||||
]
|
|
||||||
```
|
|
||||||
|
|
||||||
### get_employee
|
|
||||||
|
|
||||||
Назначение: получить одну карточку сотрудника.
|
|
||||||
|
|
||||||
Аргументы:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"profile_id_or_url": "avsergeev"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Поиск выполняется по:
|
|
||||||
|
|
||||||
- `profile_key`
|
|
||||||
- `profile_id`
|
|
||||||
- точному `canonical_url`
|
|
||||||
- частичному совпадению `canonical_url`
|
|
||||||
|
|
||||||
Возвращает полный employee payload с `data`.
|
|
||||||
|
|
||||||
Если сотрудник не найден:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"error": "not_found"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### list_employee_publications
|
|
||||||
|
|
||||||
Назначение: вернуть публикации сотрудника. Если есть нормализованные строки в `employee_publications`, tool возвращает детальные публикационные данные: авторов, DOI, аннотацию, описание, citation text, год, тип, язык, статус и ссылки. Если детальная таблица еще не заполнена, tool использует старый fallback из `employees.current_data.sections[].publications`.
|
|
||||||
|
|
||||||
Аргументы:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"profile_id_or_url": "avsergeev"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Поиск сотрудника выполняется так же, как в `get_employee`: по `profile_key`, `profile_id`, точному или частичному `canonical_url`.
|
|
||||||
|
|
||||||
Порядок источников:
|
|
||||||
|
|
||||||
- сначала `employee_publications`, отсортированные по году, названию и внутреннему id;
|
|
||||||
- если записей нет, секции `current_data.sections` с `type = "publications"` и массивами `publications`.
|
|
||||||
|
|
||||||
Ответ:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"employee": {
|
|
||||||
"profile_key": "org_person:803294906",
|
|
||||||
"profile_id": "803294906",
|
|
||||||
"full_name": "Борисов Сергей Петрович",
|
|
||||||
"status": "active",
|
|
||||||
"canonical_url": "https://www.hse.ru/org/persons/803294906",
|
|
||||||
"last_seen_at": "2026-05-14T10:00:00+00:00",
|
|
||||||
"dismissed_at": null
|
|
||||||
},
|
|
||||||
"items": [
|
|
||||||
{
|
|
||||||
"id": "888959076",
|
|
||||||
"publication_id": "888959076",
|
|
||||||
"title": "Название публикации",
|
|
||||||
"text": "Краткое описание или citation",
|
|
||||||
"url": "https://publications.hse.ru/view/888959076",
|
|
||||||
"year": 2023,
|
|
||||||
"type": "ARTICLE",
|
|
||||||
"publication_type": "ARTICLE",
|
|
||||||
"language": "ru",
|
|
||||||
"status": 1,
|
|
||||||
"doi_url": "https://doi.org/10.53921/18195822_2023_23_4_624",
|
|
||||||
"other_url": "https://example.test",
|
|
||||||
"document_url": "https://example.test/file.pdf",
|
|
||||||
"citation_text": "Авторы. Название публикации // Журнал. 2023.",
|
|
||||||
"annotation": {
|
|
||||||
"ru": "Аннотация",
|
|
||||||
"en": "Abstract"
|
|
||||||
},
|
|
||||||
"description": {
|
|
||||||
"main": "Авторы. Название публикации // Журнал. 2023."
|
|
||||||
},
|
|
||||||
"authors": [
|
|
||||||
{
|
|
||||||
"id": "803294906",
|
|
||||||
"href": "https://www.hse.ru/org/persons/803294906",
|
|
||||||
"title_ru": "Борисов С. П.",
|
|
||||||
"title_en": "",
|
|
||||||
"reverse_title_ru": "С. П. Борисов",
|
|
||||||
"reverse_title_en": "",
|
|
||||||
"alt_name": "S. P. Borisov",
|
|
||||||
"other_name": null,
|
|
||||||
"is_current_employee": true
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
В fallback-режиме из `current_data` старые элементы могут содержать только базовые поля `title`, `text`, `url` и `id`.
|
|
||||||
|
|
||||||
Если сотрудник не найден:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"items": []
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Если сотрудник найден, но публикаций нет:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"employee": {},
|
|
||||||
"items": []
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### list_employee_courses
|
|
||||||
|
|
||||||
Назначение: вернуть курсы преподавания сотрудника из распарсенных секций профиля.
|
|
||||||
|
|
||||||
Аргументы:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"profile_id_or_url": "avsergeev"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Сервис ищет секции `current_data.sections` с `type = "courses_by_year"` и объединяет массивы `courses`.
|
|
||||||
|
|
||||||
Ответ:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"employee": {},
|
|
||||||
"items": [
|
|
||||||
{
|
|
||||||
"title": "Название курса",
|
|
||||||
"url": "https://..."
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Если сотрудник или данные профиля отсутствуют:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"items": []
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### get_crawl_status
|
|
||||||
|
|
||||||
Назначение: вернуть последний запуск парсинга.
|
|
||||||
|
|
||||||
Аргументы: отсутствуют.
|
|
||||||
|
|
||||||
Ответ:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"id": 123,
|
|
||||||
"status": "completed",
|
|
||||||
"source_url": "https://miem.hse.ru/persons",
|
|
||||||
"started_at": "2026-05-14T10:00:00+00:00",
|
|
||||||
"finished_at": "2026-05-14T10:10:00+00:00",
|
|
||||||
"found_count": 100,
|
|
||||||
"parsed_count": 98,
|
|
||||||
"error_count": 2,
|
|
||||||
"dismissed_count": 1
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Если запусков еще не было:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"status": "never_run"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### get_crawl_run_details
|
|
||||||
|
|
||||||
Назначение: вернуть детальную информацию по конкретному запуску парсинга: summary, изменения сотрудников и ошибки.
|
|
||||||
|
|
||||||
Аргументы:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"run_id": 123
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Ответ:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"id": 123,
|
|
||||||
"source_url": "https://miem.hse.ru/persons",
|
|
||||||
"status": "completed",
|
|
||||||
"status_display": "Завершен",
|
|
||||||
"started_at": "2026-05-14T10:00:00+00:00",
|
|
||||||
"finished_at": "2026-05-14T10:10:00+00:00",
|
|
||||||
"started_display": "14.05.2026 13:00",
|
|
||||||
"finished_display": "14.05.2026 13:10",
|
|
||||||
"found_count": 100,
|
|
||||||
"parsed_count": 98,
|
|
||||||
"new_count": 3,
|
|
||||||
"error_count": 2,
|
|
||||||
"dismissed_count": 1,
|
|
||||||
"processed_count": 100,
|
|
||||||
"progress_percent": 100.0,
|
|
||||||
"message": null,
|
|
||||||
"changes_detail_available": true,
|
|
||||||
"changes": {
|
|
||||||
"new": [],
|
|
||||||
"missing_from_source": [],
|
|
||||||
"dismissed": []
|
|
||||||
},
|
|
||||||
"errors": []
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Если запуск не найден:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"error": "not_found"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Примеры curl
|
|
||||||
|
|
||||||
Список tools:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
curl http://localhost:8000/mcp \
|
|
||||||
-H "Content-Type: application/json" \
|
|
||||||
-d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}'
|
|
||||||
```
|
|
||||||
|
|
||||||
Поиск сотрудника:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
curl http://localhost:8000/mcp \
|
|
||||||
-H "Content-Type: application/json" \
|
|
||||||
-d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"search_employees","arguments":{"query":"Сергеев","limit":5}}}'
|
|
||||||
```
|
|
||||||
|
|
||||||
Полная синхронизация:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
curl http://localhost:8000/mcp \
|
|
||||||
-H "Content-Type: application/json" \
|
|
||||||
-d '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"sync_employees","arguments":{"include_data":false}}}'
|
|
||||||
```
|
|
||||||
|
|
||||||
Delta-синхронизация:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
curl http://localhost:8000/mcp \
|
|
||||||
-H "Content-Type: application/json" \
|
|
||||||
-d '{"jsonrpc":"2.0","id":4,"method":"tools/call","params":{"name":"sync_employees","arguments":{"client_hash":"known-sha256","include_data":true}}}'
|
|
||||||
```
|
|
||||||
|
|
||||||
## Как MCP используется клиентом
|
|
||||||
|
|
||||||
1. Клиент вызывает `initialize` и проверяет `protocolVersion`.
|
|
||||||
2. Клиент вызывает `tools/list`, чтобы получить актуальный список tools и input schemas.
|
|
||||||
3. Для поиска и точечных запросов клиент вызывает `tools/call` с `search_employees`, `get_employee`, `list_employee_publications`, `list_employee_courses`, `get_crawl_status` или `get_crawl_run_details`.
|
|
||||||
4. Для локального кэша клиент вызывает `get_service_info` или `sync_employees`.
|
|
||||||
5. Клиент хранит последний `dataset.hash`.
|
|
||||||
6. При следующей синхронизации клиент передает hash как `client_hash`.
|
|
||||||
7. Сервер возвращает пустую delta, delta с изменениями или полный snapshot, если hash неизвестен.
|
|
||||||
|
|
||||||
## Важные особенности реализации
|
|
||||||
|
|
||||||
- MCP endpoint read-only: tools не запускают парсинг и не меняют сотрудников напрямую.
|
|
||||||
- `get_service_info` и `sync_employees` могут создать новую запись `dataset_versions`, если состояние сотрудников изменилось и новой версии еще нет.
|
|
||||||
- Все tool payloads возвращаются как JSON-строка внутри `content[0].text`.
|
|
||||||
- `search_employees` ищет через `ilike` по `full_name` и `canonical_url`.
|
|
||||||
- `get_employee` допускает частичный URL, поэтому строка `133709486` может найти `https://www.hse.ru/org/persons/133709486`.
|
|
||||||
- Временные значения сериализуются через `isoformat()`, display-поля для админских payload формируются в часовом поясе `Europe/Moscow`.
|
|
||||||
@@ -1,10 +1,11 @@
|
|||||||
# MIEM Employees Server
|
# MIEM Employees Server
|
||||||
|
|
||||||
Сервис собирает сотрудников МИЭМ с сайта ВШЭ, хранит карточки и историю обновлений в Postgres, показывает минимальную админку и отдает read-only MCP endpoint для ИИ-агентов.
|
Сервис собирает сотрудников МИЭМ с сайта ВШЭ, хранит карточки и историю обновлений в Postgres и показывает минимальную админку.
|
||||||
|
|
||||||
## Архитектура
|
## Архитектура
|
||||||
|
|
||||||
- `api`: FastAPI, REST API, HTML-админка, MCP endpoint и healthcheck.
|
- `api`: FastAPI, REST API, HTML-админка и healthcheck.
|
||||||
|
- `worker`: weekly scheduler, который запускает парсинг по `CRAWL_CRON`.
|
||||||
- `postgres`: основная БД.
|
- `postgres`: основная БД.
|
||||||
|
|
||||||
Парсер использует фиксированный источник сотрудников, по умолчанию `https://miem.hse.ru/persons`. Для каждой карточки сохраняются ФИО, должности, год начала работы, контакты, идентификаторы, вкладки профиля, секции, публикации, курсы, ВКР, новости, JSON-снапшот и сжатый HTML-снапшот. Детальные публикации дополнительно нормализуются в отдельную таблицу `employee_publications`, а новости из блока «В новостях» — в `employee_news_links`. Ссылки обходятся только из меню профиля самого сотрудника (`person-menu`), например `#sci`, `#teaching`, `#main`.
|
Парсер использует фиксированный источник сотрудников, по умолчанию `https://miem.hse.ru/persons`. Для каждой карточки сохраняются ФИО, должности, год начала работы, контакты, идентификаторы, вкладки профиля, секции, публикации, курсы, ВКР, новости, JSON-снапшот и сжатый HTML-снапшот. Детальные публикации дополнительно нормализуются в отдельную таблицу `employee_publications`, а новости из блока «В новостях» — в `employee_news_links`. Ссылки обходятся только из меню профиля самого сотрудника (`person-menu`), например `#sci`, `#teaching`, `#main`.
|
||||||
@@ -20,7 +21,8 @@ cp .env.example .env
|
|||||||
Основные настройки:
|
Основные настройки:
|
||||||
|
|
||||||
- `DATABASE_URL`: строка подключения SQLAlchemy.
|
- `DATABASE_URL`: строка подключения SQLAlchemy.
|
||||||
- `SOURCE_URL`: список сотрудников МИЭМ.
|
- `SOURCE_URL`: список сотрудников МИЭМ.
|
||||||
|
- `CRAWL_CRON`: расписание в формате crontab, по умолчанию `0 3 * * 1`.
|
||||||
- `CRAWL_LIMIT`: опциональный лимит профилей для тестового запуска.
|
- `CRAWL_LIMIT`: опциональный лимит профилей для тестового запуска.
|
||||||
- `ADMIN_USERNAME`, `ADMIN_PASSWORD`: логин и пароль админки.
|
- `ADMIN_USERNAME`, `ADMIN_PASSWORD`: логин и пароль админки.
|
||||||
- `SESSION_SECRET`: секрет подписи cookie.
|
- `SESSION_SECRET`: секрет подписи cookie.
|
||||||
@@ -80,7 +82,7 @@ docker compose up -d --build --remove-orphans
|
|||||||
|
|
||||||
## Парсинг
|
## Парсинг
|
||||||
|
|
||||||
Ручной запуск доступен в админке на `Dashboard` и странице `Runs` или через REST:
|
Worker запускает обход по `CRAWL_CRON`. Ручной запуск также доступен в админке на `Dashboard` и странице `Runs` или через REST:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
curl -X POST http://localhost:8000/api/crawl-runs --cookie "miem_admin_session=..."
|
curl -X POST http://localhost:8000/api/crawl-runs --cookie "miem_admin_session=..."
|
||||||
@@ -105,41 +107,13 @@ curl -X POST http://localhost:8000/api/crawl-runs --cookie "miem_admin_session=.
|
|||||||
|
|
||||||
Во время выполнения парсинга `found_count`, `parsed_count`, `skipped_count` и `error_count` обновляются в базе. Админка опрашивает `/api/crawl-runs/latest` и показывает прогресс как `(parsed_count + skipped_count + error_count) / found_count`.
|
Во время выполнения парсинга `found_count`, `parsed_count`, `skipped_count` и `error_count` обновляются в базе. Админка опрашивает `/api/crawl-runs/latest` и показывает прогресс как `(parsed_count + skipped_count + error_count) / found_count`.
|
||||||
|
|
||||||
## MCP
|
## Обслуживание
|
||||||
|
|
||||||
Endpoint: `POST /mcp`, без авторизации на уровне приложения.
|
|
||||||
|
|
||||||
Поддерживаемые tools:
|
|
||||||
|
|
||||||
- `get_service_info()`
|
|
||||||
- `sync_employees(client_hash?, include_data?)`
|
|
||||||
- `search_employees(query, status?, limit?)`
|
|
||||||
- `get_employee(profile_id_or_url)`
|
|
||||||
- `list_employee_publications(profile_id_or_url)` — публикации сотрудника; при наличии данных из `employee_publications` возвращает авторов, DOI, аннотацию, описание, citation text, год, тип, язык, статус и ссылку HSE Publications.
|
|
||||||
- `list_employee_courses(profile_id_or_url)`
|
|
||||||
- `get_crawl_status()`
|
|
||||||
- `get_crawl_run_details(run_id)`
|
|
||||||
|
|
||||||
`get_service_info` возвращает метаданные сервиса, список tools и текущую версию набора сотрудников. `sync_employees` отдает полный snapshot или delta по `client_hash`; checksum набора строится по сотрудникам, их статусам и текущим checksums. Ответы tools возвращаются как JSON-строка внутри MCP `content[0].text`.
|
|
||||||
|
|
||||||
Новости сотрудника отдельной MCP tool не имеют: они доступны в `get_employee(...).data.sections` и `sync_employees(include_data=true)` как секция `type = "news"` с массивом `news_links`.
|
|
||||||
|
|
||||||
Пример локального запроса списка tools:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
curl http://localhost:8000/mcp \
|
docker compose logs -f api
|
||||||
-H "Content-Type: application/json" \
|
docker compose logs -f worker
|
||||||
-d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}'
|
|
||||||
```
|
|
||||||
|
|
||||||
Если MCP нужно ограничить, делайте это на сетевом уровне: localhost binding, VPN, firewall, reverse proxy или другой внешний контур доступа.
|
|
||||||
|
|
||||||
## Обслуживание
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docker compose logs -f api
|
|
||||||
docker compose exec postgres pg_dump -U miem miem_workers > backup.sql
|
docker compose exec postgres pg_dump -U miem miem_workers > backup.sql
|
||||||
docker compose down
|
docker compose down
|
||||||
```
|
```
|
||||||
|
|
||||||
Версия сервиса: `0.7.5`. Админка всегда показывает версии backend и frontend в footer.
|
Версия сервиса: `0.7.7`. Админка всегда показывает версии backend и frontend в footer.
|
||||||
@@ -4,14 +4,12 @@ from fastapi.staticfiles import StaticFiles
|
|||||||
from app.admin import router as admin_router
|
from app.admin import router as admin_router
|
||||||
from app.api import router as api_router
|
from app.api import router as api_router
|
||||||
from app.db import init_db
|
from app.db import init_db
|
||||||
from app.mcp import router as mcp_router
|
|
||||||
from app.version import BACKEND_VERSION
|
from app.version import BACKEND_VERSION
|
||||||
|
|
||||||
app = FastAPI(title="MIEM Employees", version=BACKEND_VERSION)
|
app = FastAPI(title="MIEM Employees", version=BACKEND_VERSION)
|
||||||
app.mount("/static", StaticFiles(directory="app/static"), name="static")
|
app.mount("/static", StaticFiles(directory="app/static"), name="static")
|
||||||
app.include_router(api_router)
|
app.include_router(api_router)
|
||||||
app.include_router(admin_router)
|
app.include_router(admin_router)
|
||||||
app.include_router(mcp_router)
|
|
||||||
|
|
||||||
|
|
||||||
@app.on_event("startup")
|
@app.on_event("startup")
|
||||||
|
|||||||
-262
@@ -1,262 +0,0 @@
|
|||||||
import json
|
|
||||||
|
|
||||||
from fastapi import APIRouter, Depends, Request
|
|
||||||
from sqlalchemy import desc, or_, select
|
|
||||||
from sqlalchemy.orm import Session
|
|
||||||
|
|
||||||
from app.db import get_db
|
|
||||||
from app.models import CrawlRun, Employee, EmployeePublication
|
|
||||||
from app.services.admin_data import run_detail_payload
|
|
||||||
from app.services.dataset_versions import service_info_payload, sync_employees_payload
|
|
||||||
from app.version import BACKEND_VERSION
|
|
||||||
|
|
||||||
router = APIRouter(prefix="/mcp")
|
|
||||||
PROTOCOL_VERSION = "2024-11-05"
|
|
||||||
SERVICE_NAME = "miem-employees"
|
|
||||||
|
|
||||||
|
|
||||||
TOOLS = [
|
|
||||||
{
|
|
||||||
"name": "get_service_info",
|
|
||||||
"description": "Return service metadata, supported tools, and current dataset version.",
|
|
||||||
"inputSchema": {"type": "object", "properties": {}},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"name": "sync_employees",
|
|
||||||
"description": "Synchronize employees by dataset hash. Returns a full snapshot or a delta from client_hash.",
|
|
||||||
"inputSchema": {
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"client_hash": {"type": "string"},
|
|
||||||
"include_data": {"type": "boolean", "default": True},
|
|
||||||
},
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"name": "search_employees",
|
|
||||||
"description": "Search MIEM employees by name or profile URL.",
|
|
||||||
"inputSchema": {
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"query": {"type": "string"},
|
|
||||||
"status": {"type": "string", "enum": ["active", "dismissed"]},
|
|
||||||
"limit": {"type": "integer", "default": 20},
|
|
||||||
},
|
|
||||||
"required": ["query"],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"name": "get_employee",
|
|
||||||
"description": "Get one employee by profile id, profile key, or canonical URL.",
|
|
||||||
"inputSchema": {"type": "object", "properties": {"profile_id_or_url": {"type": "string"}}, "required": ["profile_id_or_url"]},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"name": "list_employee_publications",
|
|
||||||
"description": (
|
|
||||||
"List employee publications with detailed fields when available: authors, DOI URL, annotation, "
|
|
||||||
"description, citation text, year, publication type, language, status, and HSE Publications URL."
|
|
||||||
),
|
|
||||||
"inputSchema": {"type": "object", "properties": {"profile_id_or_url": {"type": "string"}}, "required": ["profile_id_or_url"]},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"name": "list_employee_courses",
|
|
||||||
"description": "List teaching courses parsed from an employee profile.",
|
|
||||||
"inputSchema": {"type": "object", "properties": {"profile_id_or_url": {"type": "string"}}, "required": ["profile_id_or_url"]},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"name": "get_crawl_status",
|
|
||||||
"description": "Return the latest crawl run status.",
|
|
||||||
"inputSchema": {"type": "object", "properties": {}},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"name": "get_crawl_run_details",
|
|
||||||
"description": "Return detailed employee changes and errors for one crawl run.",
|
|
||||||
"inputSchema": {
|
|
||||||
"type": "object",
|
|
||||||
"properties": {"run_id": {"type": "integer"}},
|
|
||||||
"required": ["run_id"],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
]
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("")
|
|
||||||
async def mcp_http(
|
|
||||||
request: Request,
|
|
||||||
db: Session = Depends(get_db),
|
|
||||||
) -> dict:
|
|
||||||
payload = await request.json()
|
|
||||||
method = payload.get("method")
|
|
||||||
request_id = payload.get("id")
|
|
||||||
params = payload.get("params") or {}
|
|
||||||
|
|
||||||
try:
|
|
||||||
if method == "initialize":
|
|
||||||
result = {
|
|
||||||
"protocolVersion": PROTOCOL_VERSION,
|
|
||||||
"serverInfo": {"name": SERVICE_NAME, "version": BACKEND_VERSION},
|
|
||||||
"capabilities": {"tools": {}},
|
|
||||||
}
|
|
||||||
elif method == "tools/list":
|
|
||||||
result = {"tools": TOOLS}
|
|
||||||
elif method == "tools/call":
|
|
||||||
result = _call_tool(db, params.get("name"), params.get("arguments") or {})
|
|
||||||
else:
|
|
||||||
return {"jsonrpc": "2.0", "id": request_id, "error": {"code": -32601, "message": "Method not found"}}
|
|
||||||
return {"jsonrpc": "2.0", "id": request_id, "result": result}
|
|
||||||
except Exception as exc:
|
|
||||||
return {"jsonrpc": "2.0", "id": request_id, "error": {"code": -32000, "message": str(exc)}}
|
|
||||||
|
|
||||||
|
|
||||||
def _call_tool(db: Session, name: str, arguments: dict) -> dict:
|
|
||||||
if name == "get_service_info":
|
|
||||||
return _tool_response(
|
|
||||||
service_info_payload(
|
|
||||||
db,
|
|
||||||
tools=TOOLS,
|
|
||||||
service_name=SERVICE_NAME,
|
|
||||||
backend_version=BACKEND_VERSION,
|
|
||||||
protocol_version=PROTOCOL_VERSION,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
if name == "sync_employees":
|
|
||||||
return _tool_response(
|
|
||||||
sync_employees_payload(
|
|
||||||
db,
|
|
||||||
client_hash=arguments.get("client_hash"),
|
|
||||||
include_data=bool(arguments.get("include_data", True)),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
if name == "search_employees":
|
|
||||||
return _tool_response(_search_employees(db, arguments))
|
|
||||||
if name == "get_employee":
|
|
||||||
employee = _find_employee(db, arguments["profile_id_or_url"])
|
|
||||||
return _tool_response(_employee_payload(employee) if employee else {"error": "not_found"})
|
|
||||||
if name == "list_employee_publications":
|
|
||||||
employee = _find_employee(db, arguments["profile_id_or_url"])
|
|
||||||
return _tool_response(_collect_section_items(employee, "publications"))
|
|
||||||
if name == "list_employee_courses":
|
|
||||||
employee = _find_employee(db, arguments["profile_id_or_url"])
|
|
||||||
return _tool_response(_collect_section_items(employee, "courses_by_year"))
|
|
||||||
if name == "get_crawl_status":
|
|
||||||
run = db.scalar(select(CrawlRun).order_by(desc(CrawlRun.started_at)).limit(1))
|
|
||||||
return _tool_response(_run_payload(run) if run else {"status": "never_run"})
|
|
||||||
if name == "get_crawl_run_details":
|
|
||||||
run = db.get(CrawlRun, int(arguments["run_id"]))
|
|
||||||
return _tool_response(run_detail_payload(db, run) if run else {"error": "not_found"})
|
|
||||||
raise ValueError(f"Unknown tool: {name}")
|
|
||||||
|
|
||||||
|
|
||||||
def _search_employees(db: Session, arguments: dict) -> list[dict]:
|
|
||||||
query = arguments.get("query", "")
|
|
||||||
limit = min(int(arguments.get("limit") or 20), 100)
|
|
||||||
stmt = select(Employee)
|
|
||||||
if arguments.get("status"):
|
|
||||||
stmt = stmt.where(Employee.status == arguments["status"])
|
|
||||||
if query:
|
|
||||||
pattern = f"%{query}%"
|
|
||||||
stmt = stmt.where(or_(Employee.full_name.ilike(pattern), Employee.canonical_url.ilike(pattern)))
|
|
||||||
employees = db.scalars(stmt.order_by(Employee.full_name).limit(limit)).all()
|
|
||||||
return [_employee_payload(employee, include_data=False) for employee in employees]
|
|
||||||
|
|
||||||
|
|
||||||
def _find_employee(db: Session, value: str) -> Employee | None:
|
|
||||||
pattern = value.strip()
|
|
||||||
stmt = select(Employee).where(
|
|
||||||
or_(
|
|
||||||
Employee.profile_key == pattern,
|
|
||||||
Employee.profile_id == pattern,
|
|
||||||
Employee.canonical_url == pattern,
|
|
||||||
Employee.canonical_url.ilike(f"%{pattern}%"),
|
|
||||||
)
|
|
||||||
)
|
|
||||||
return db.scalar(stmt.limit(1))
|
|
||||||
|
|
||||||
|
|
||||||
def _collect_section_items(employee: Employee | None, section_type: str) -> dict:
|
|
||||||
if not employee:
|
|
||||||
return {"items": []}
|
|
||||||
if section_type == "publications":
|
|
||||||
publications = _stored_publications(employee)
|
|
||||||
if publications:
|
|
||||||
return {"employee": _employee_payload(employee, include_data=False), "items": publications}
|
|
||||||
if not employee.current_data:
|
|
||||||
return {"employee": _employee_payload(employee, include_data=False), "items": []}
|
|
||||||
items = []
|
|
||||||
for section in employee.current_data.get("sections") or []:
|
|
||||||
if section.get("type") != section_type:
|
|
||||||
continue
|
|
||||||
if section_type == "publications":
|
|
||||||
items.extend(section.get("publications") or [])
|
|
||||||
elif section_type == "courses_by_year":
|
|
||||||
items.extend(section.get("courses") or [])
|
|
||||||
return {"employee": _employee_payload(employee, include_data=False), "items": items}
|
|
||||||
|
|
||||||
|
|
||||||
def _stored_publications(employee: Employee) -> list[dict]:
|
|
||||||
return [_publication_payload(publication) for publication in sorted(employee.publications, key=_publication_sort_key)]
|
|
||||||
|
|
||||||
|
|
||||||
def _publication_sort_key(publication: EmployeePublication) -> tuple:
|
|
||||||
return (publication.year or 0, publication.title or "", publication.id)
|
|
||||||
|
|
||||||
|
|
||||||
def _publication_payload(publication: EmployeePublication) -> dict:
|
|
||||||
text = publication.citation_text or publication.title
|
|
||||||
payload = {
|
|
||||||
"id": publication.publication_id,
|
|
||||||
"publication_id": publication.publication_id,
|
|
||||||
"title": publication.title,
|
|
||||||
"text": text,
|
|
||||||
"url": publication.url,
|
|
||||||
}
|
|
||||||
optional = {
|
|
||||||
"year": publication.year,
|
|
||||||
"type": publication.publication_type,
|
|
||||||
"publication_type": publication.publication_type,
|
|
||||||
"language": publication.language,
|
|
||||||
"status": publication.status,
|
|
||||||
"doi_url": publication.doi_url,
|
|
||||||
"other_url": publication.other_url,
|
|
||||||
"document_url": publication.document_url,
|
|
||||||
"citation_text": publication.citation_text,
|
|
||||||
"annotation": publication.annotation,
|
|
||||||
"description": publication.description,
|
|
||||||
"authors": publication.authors,
|
|
||||||
}
|
|
||||||
payload.update({key: value for key, value in optional.items() if value not in (None, [], {})})
|
|
||||||
return payload
|
|
||||||
|
|
||||||
|
|
||||||
def _employee_payload(employee: Employee, include_data: bool = True) -> dict:
|
|
||||||
payload = {
|
|
||||||
"profile_key": employee.profile_key,
|
|
||||||
"profile_id": employee.profile_id,
|
|
||||||
"full_name": employee.full_name,
|
|
||||||
"status": employee.status,
|
|
||||||
"canonical_url": employee.canonical_url,
|
|
||||||
"last_seen_at": employee.last_seen_at.isoformat() if employee.last_seen_at else None,
|
|
||||||
"dismissed_at": employee.dismissed_at.isoformat() if employee.dismissed_at else None,
|
|
||||||
}
|
|
||||||
if include_data:
|
|
||||||
payload["data"] = employee.current_data
|
|
||||||
return payload
|
|
||||||
|
|
||||||
|
|
||||||
def _run_payload(run: CrawlRun) -> dict:
|
|
||||||
return {
|
|
||||||
"id": run.id,
|
|
||||||
"status": run.status,
|
|
||||||
"source_url": run.source_url,
|
|
||||||
"started_at": run.started_at.isoformat() if run.started_at else None,
|
|
||||||
"finished_at": run.finished_at.isoformat() if run.finished_at else None,
|
|
||||||
"found_count": run.found_count,
|
|
||||||
"parsed_count": run.parsed_count,
|
|
||||||
"skipped_count": run.skipped_count,
|
|
||||||
"error_count": run.error_count,
|
|
||||||
"dismissed_count": run.dismissed_count,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _tool_response(data: object) -> dict:
|
|
||||||
return {"content": [{"type": "text", "text": json.dumps(data, ensure_ascii=False, default=str)}]}
|
|
||||||
@@ -162,7 +162,6 @@ class CrawlRun(Base):
|
|||||||
message: Mapped[str | None] = mapped_column(Text)
|
message: Mapped[str | None] = mapped_column(Text)
|
||||||
|
|
||||||
employee_changes: Mapped[list["CrawlRunEmployeeChange"]] = relationship(back_populates="crawl_run")
|
employee_changes: Mapped[list["CrawlRunEmployeeChange"]] = relationship(back_populates="crawl_run")
|
||||||
dataset_versions: Mapped[list["DatasetVersion"]] = relationship(back_populates="crawl_run")
|
|
||||||
|
|
||||||
|
|
||||||
class CrawlRunEmployeeChange(Base):
|
class CrawlRunEmployeeChange(Base):
|
||||||
@@ -242,42 +241,3 @@ class ParseResourceCache(Base):
|
|||||||
body_snapshot: Mapped[bytes] = mapped_column(LargeBinary, nullable=False)
|
body_snapshot: Mapped[bytes] = mapped_column(LargeBinary, nullable=False)
|
||||||
parser_version: Mapped[str | None] = mapped_column(String(32))
|
parser_version: Mapped[str | None] = mapped_column(String(32))
|
||||||
fetched_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow, nullable=False)
|
fetched_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow, nullable=False)
|
||||||
|
|
||||||
|
|
||||||
class DatasetVersion(Base):
|
|
||||||
__tablename__ = "dataset_versions"
|
|
||||||
__table_args__ = (
|
|
||||||
UniqueConstraint("hash", name="uq_dataset_versions_hash"),
|
|
||||||
Index("ix_dataset_versions_created_at", "created_at"),
|
|
||||||
)
|
|
||||||
|
|
||||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
|
||||||
hash: Mapped[str] = mapped_column(String(64), nullable=False)
|
|
||||||
previous_hash: Mapped[str | None] = mapped_column(String(64))
|
|
||||||
crawl_run_id: Mapped[int | None] = mapped_column(ForeignKey("crawl_runs.id"))
|
|
||||||
employee_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
|
|
||||||
active_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
|
|
||||||
dismissed_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
|
|
||||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow, nullable=False)
|
|
||||||
|
|
||||||
crawl_run: Mapped[CrawlRun | None] = relationship(back_populates="dataset_versions")
|
|
||||||
items: Mapped[list["DatasetVersionItem"]] = relationship(back_populates="dataset_version", cascade="all, delete-orphan")
|
|
||||||
|
|
||||||
|
|
||||||
class DatasetVersionItem(Base):
|
|
||||||
__tablename__ = "dataset_version_items"
|
|
||||||
__table_args__ = (
|
|
||||||
UniqueConstraint("dataset_version_id", "profile_key", name="uq_dataset_version_items_version_profile"),
|
|
||||||
Index("ix_dataset_version_items_hash", "dataset_version_id"),
|
|
||||||
Index("ix_dataset_version_items_profile_key", "profile_key"),
|
|
||||||
)
|
|
||||||
|
|
||||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
|
||||||
dataset_version_id: Mapped[int] = mapped_column(ForeignKey("dataset_versions.id"), nullable=False)
|
|
||||||
profile_key: Mapped[str] = mapped_column(String(255), nullable=False)
|
|
||||||
employee_id: Mapped[int | None] = mapped_column(ForeignKey("employees.id"))
|
|
||||||
status: Mapped[str] = mapped_column(String(32), nullable=False)
|
|
||||||
checksum: Mapped[str] = mapped_column(String(64), nullable=False)
|
|
||||||
|
|
||||||
dataset_version: Mapped[DatasetVersion] = relationship(back_populates="items")
|
|
||||||
employee: Mapped[Employee | None] = relationship()
|
|
||||||
@@ -26,7 +26,6 @@ from app.parser.collector import collect_profile_links
|
|||||||
from app.parser.profile import parse_person_profile
|
from app.parser.profile import parse_person_profile
|
||||||
from app.parser.profile_url import profile_key
|
from app.parser.profile_url import profile_key
|
||||||
from app.services.academic_degrees import academic_degrees
|
from app.services.academic_degrees import academic_degrees
|
||||||
from app.services.dataset_versions import get_or_create_current_version
|
|
||||||
from app.services.resource_cache import ResourceCache
|
from app.services.resource_cache import ResourceCache
|
||||||
|
|
||||||
HEADERS = {
|
HEADERS = {
|
||||||
@@ -102,7 +101,6 @@ def run_crawl(db: Session, settings: Settings) -> CrawlRun:
|
|||||||
max_auto_dismissals=settings.max_auto_dismissals_per_run,
|
max_auto_dismissals=settings.max_auto_dismissals_per_run,
|
||||||
)
|
)
|
||||||
run.status = "completed"
|
run.status = "completed"
|
||||||
get_or_create_current_version(db, crawl_run_id=run.id)
|
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
run.status = "failed"
|
run.status = "failed"
|
||||||
run.message = str(exc)
|
run.message = str(exc)
|
||||||
@@ -146,7 +144,6 @@ def refresh_dismissed_status(db: Session, settings: Settings) -> CrawlRun:
|
|||||||
)
|
)
|
||||||
run.parsed_count += 1
|
run.parsed_count += 1
|
||||||
run.status = "completed"
|
run.status = "completed"
|
||||||
get_or_create_current_version(db, crawl_run_id=run.id)
|
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
run.status = "failed"
|
run.status = "failed"
|
||||||
run.error_count = 1
|
run.error_count = 1
|
||||||
@@ -194,7 +191,6 @@ def refresh_employee(db: Session, employee: Employee, settings: Settings) -> Cra
|
|||||||
else:
|
else:
|
||||||
run.skipped_count = 1
|
run.skipped_count = 1
|
||||||
run.status = "completed"
|
run.status = "completed"
|
||||||
get_or_create_current_version(db, crawl_run_id=run.id)
|
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
run.status = "failed"
|
run.status = "failed"
|
||||||
run.error_count = 1
|
run.error_count = 1
|
||||||
|
|||||||
@@ -1,227 +0,0 @@
|
|||||||
import hashlib
|
|
||||||
import json
|
|
||||||
from dataclasses import dataclass
|
|
||||||
|
|
||||||
from sqlalchemy import desc, select
|
|
||||||
from sqlalchemy.orm import Session
|
|
||||||
|
|
||||||
from app.models import DatasetVersion, DatasetVersionItem, Employee
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class EmployeeMarker:
|
|
||||||
profile_key: str
|
|
||||||
employee_id: int | None
|
|
||||||
status: str
|
|
||||||
checksum: str
|
|
||||||
|
|
||||||
|
|
||||||
def get_or_create_current_version(db: Session, *, crawl_run_id: int | None = None) -> DatasetVersion:
|
|
||||||
employees = db.scalars(select(Employee).order_by(Employee.profile_key)).all()
|
|
||||||
markers = [_employee_marker(employee) for employee in employees]
|
|
||||||
dataset_hash = _dataset_hash(markers)
|
|
||||||
latest = get_latest_version(db)
|
|
||||||
if latest and latest.hash == dataset_hash:
|
|
||||||
return latest
|
|
||||||
|
|
||||||
active_count = sum(1 for marker in markers if marker.status == "active")
|
|
||||||
dismissed_count = sum(1 for marker in markers if marker.status == "dismissed")
|
|
||||||
version = DatasetVersion(
|
|
||||||
hash=dataset_hash,
|
|
||||||
previous_hash=latest.hash if latest else None,
|
|
||||||
crawl_run_id=crawl_run_id,
|
|
||||||
employee_count=len(markers),
|
|
||||||
active_count=active_count,
|
|
||||||
dismissed_count=dismissed_count,
|
|
||||||
)
|
|
||||||
db.add(version)
|
|
||||||
db.flush()
|
|
||||||
for marker in markers:
|
|
||||||
db.add(
|
|
||||||
DatasetVersionItem(
|
|
||||||
dataset_version_id=version.id,
|
|
||||||
profile_key=marker.profile_key,
|
|
||||||
employee_id=marker.employee_id,
|
|
||||||
status=marker.status,
|
|
||||||
checksum=marker.checksum,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
db.flush()
|
|
||||||
return version
|
|
||||||
|
|
||||||
|
|
||||||
def get_latest_version(db: Session) -> DatasetVersion | None:
|
|
||||||
return db.scalar(select(DatasetVersion).order_by(desc(DatasetVersion.created_at), desc(DatasetVersion.id)).limit(1))
|
|
||||||
|
|
||||||
|
|
||||||
def get_version_by_hash(db: Session, dataset_hash: str | None) -> DatasetVersion | None:
|
|
||||||
if not dataset_hash:
|
|
||||||
return None
|
|
||||||
return db.scalar(select(DatasetVersion).where(DatasetVersion.hash == dataset_hash).limit(1))
|
|
||||||
|
|
||||||
|
|
||||||
def service_info_payload(db: Session, *, tools: list[dict], service_name: str, backend_version: str, protocol_version: str) -> dict:
|
|
||||||
version = get_or_create_current_version(db)
|
|
||||||
db.commit()
|
|
||||||
return {
|
|
||||||
"service_name": service_name,
|
|
||||||
"backend_version": backend_version,
|
|
||||||
"protocolVersion": protocol_version,
|
|
||||||
"tools": tools,
|
|
||||||
"dataset": _version_payload(version),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def sync_employees_payload(db: Session, *, client_hash: str | None = None, include_data: bool = True) -> dict:
|
|
||||||
current = get_or_create_current_version(db)
|
|
||||||
db.commit()
|
|
||||||
if not client_hash:
|
|
||||||
return _full_sync_payload(db, current, include_data=include_data, reason=None)
|
|
||||||
if client_hash == current.hash:
|
|
||||||
return {
|
|
||||||
"mode": "delta",
|
|
||||||
"from_hash": client_hash,
|
|
||||||
"to_hash": current.hash,
|
|
||||||
"dataset": _version_payload(current),
|
|
||||||
"changes": {"added": [], "updated": [], "dismissed": [], "removed": []},
|
|
||||||
}
|
|
||||||
|
|
||||||
previous = get_version_by_hash(db, client_hash)
|
|
||||||
if not previous:
|
|
||||||
return _full_sync_payload(db, current, include_data=include_data, reason="unknown_client_hash", from_hash=client_hash)
|
|
||||||
|
|
||||||
return _delta_sync_payload(db, previous, current, include_data=include_data)
|
|
||||||
|
|
||||||
|
|
||||||
def _full_sync_payload(
|
|
||||||
db: Session,
|
|
||||||
current: DatasetVersion,
|
|
||||||
*,
|
|
||||||
include_data: bool,
|
|
||||||
reason: str | None,
|
|
||||||
from_hash: str | None = None,
|
|
||||||
) -> dict:
|
|
||||||
employees = db.scalars(select(Employee).order_by(Employee.profile_key)).all()
|
|
||||||
payload = {
|
|
||||||
"mode": "full",
|
|
||||||
"from_hash": from_hash,
|
|
||||||
"to_hash": current.hash,
|
|
||||||
"dataset": _version_payload(current),
|
|
||||||
"items": [_employee_payload(employee, include_data=include_data) for employee in employees],
|
|
||||||
}
|
|
||||||
if reason:
|
|
||||||
payload["reason"] = reason
|
|
||||||
return payload
|
|
||||||
|
|
||||||
|
|
||||||
def _delta_sync_payload(db: Session, previous: DatasetVersion, current: DatasetVersion, *, include_data: bool) -> dict:
|
|
||||||
previous_items = _items_by_profile_key(previous)
|
|
||||||
current_items = _items_by_profile_key(current)
|
|
||||||
employees = {employee.profile_key: employee for employee in db.scalars(select(Employee)).all()}
|
|
||||||
added = []
|
|
||||||
updated = []
|
|
||||||
dismissed = []
|
|
||||||
removed = []
|
|
||||||
|
|
||||||
for profile_key, current_item in sorted(current_items.items()):
|
|
||||||
previous_item = previous_items.get(profile_key)
|
|
||||||
employee = employees.get(profile_key)
|
|
||||||
if not previous_item:
|
|
||||||
if employee:
|
|
||||||
added.append(_employee_payload(employee, include_data=include_data))
|
|
||||||
continue
|
|
||||||
if previous_item.checksum == current_item.checksum and previous_item.status == current_item.status:
|
|
||||||
continue
|
|
||||||
if current_item.status == "dismissed":
|
|
||||||
dismissed.append(_tombstone(profile_key, current_item.status, employee))
|
|
||||||
elif employee:
|
|
||||||
updated.append(_employee_payload(employee, include_data=include_data))
|
|
||||||
|
|
||||||
for profile_key, previous_item in sorted(previous_items.items()):
|
|
||||||
if profile_key not in current_items:
|
|
||||||
removed.append(_tombstone(profile_key, "removed", employees.get(profile_key), checksum=previous_item.checksum))
|
|
||||||
|
|
||||||
return {
|
|
||||||
"mode": "delta",
|
|
||||||
"from_hash": previous.hash,
|
|
||||||
"to_hash": current.hash,
|
|
||||||
"dataset": _version_payload(current),
|
|
||||||
"changes": {
|
|
||||||
"added": added,
|
|
||||||
"updated": updated,
|
|
||||||
"dismissed": dismissed,
|
|
||||||
"removed": removed,
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _items_by_profile_key(version: DatasetVersion) -> dict[str, DatasetVersionItem]:
|
|
||||||
return {item.profile_key: item for item in version.items}
|
|
||||||
|
|
||||||
|
|
||||||
def _version_payload(version: DatasetVersion) -> dict:
|
|
||||||
return {
|
|
||||||
"hash": version.hash,
|
|
||||||
"previous_hash": version.previous_hash,
|
|
||||||
"created_at": version.created_at.isoformat() if version.created_at else None,
|
|
||||||
"crawl_run_id": version.crawl_run_id,
|
|
||||||
"employee_count": version.employee_count,
|
|
||||||
"active_count": version.active_count,
|
|
||||||
"dismissed_count": version.dismissed_count,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _employee_marker(employee: Employee) -> EmployeeMarker:
|
|
||||||
return EmployeeMarker(
|
|
||||||
profile_key=employee.profile_key,
|
|
||||||
employee_id=employee.id,
|
|
||||||
status=employee.status,
|
|
||||||
checksum=employee.current_checksum or _payload_hash(employee.current_data or {}),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _dataset_hash(markers: list[EmployeeMarker]) -> str:
|
|
||||||
payload = [
|
|
||||||
{"profile_key": marker.profile_key, "status": marker.status, "checksum": marker.checksum}
|
|
||||||
for marker in sorted(markers, key=lambda item: item.profile_key)
|
|
||||||
]
|
|
||||||
return _payload_hash(payload)
|
|
||||||
|
|
||||||
|
|
||||||
def _payload_hash(value: object) -> str:
|
|
||||||
payload = json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":"), default=str)
|
|
||||||
return hashlib.sha256(payload.encode("utf-8")).hexdigest()
|
|
||||||
|
|
||||||
|
|
||||||
def _employee_payload(employee: Employee, *, include_data: bool) -> dict:
|
|
||||||
payload = {
|
|
||||||
"profile_key": employee.profile_key,
|
|
||||||
"profile_id": employee.profile_id,
|
|
||||||
"full_name": employee.full_name,
|
|
||||||
"status": employee.status,
|
|
||||||
"canonical_url": employee.canonical_url,
|
|
||||||
"last_seen_at": employee.last_seen_at.isoformat() if employee.last_seen_at else None,
|
|
||||||
"dismissed_at": employee.dismissed_at.isoformat() if employee.dismissed_at else None,
|
|
||||||
"checksum": employee.current_checksum or _payload_hash(employee.current_data or {}),
|
|
||||||
}
|
|
||||||
if include_data:
|
|
||||||
payload["data"] = employee.current_data
|
|
||||||
return payload
|
|
||||||
|
|
||||||
|
|
||||||
def _tombstone(profile_key: str, status: str, employee: Employee | None, *, checksum: str | None = None) -> dict:
|
|
||||||
payload = {
|
|
||||||
"profile_key": profile_key,
|
|
||||||
"status": status,
|
|
||||||
"checksum": checksum or (employee.current_checksum if employee else None),
|
|
||||||
}
|
|
||||||
if employee:
|
|
||||||
payload.update(
|
|
||||||
{
|
|
||||||
"profile_id": employee.profile_id,
|
|
||||||
"full_name": employee.full_name,
|
|
||||||
"canonical_url": employee.canonical_url,
|
|
||||||
"dismissed_at": employee.dismissed_at.isoformat() if employee.dismissed_at else None,
|
|
||||||
}
|
|
||||||
)
|
|
||||||
return payload
|
|
||||||
+3
-3
@@ -1,3 +1,3 @@
|
|||||||
APP_VERSION = "0.7.5"
|
APP_VERSION = "0.7.7"
|
||||||
FRONTEND_VERSION = "0.7.5"
|
FRONTEND_VERSION = "0.7.7"
|
||||||
BACKEND_VERSION = "0.7.5"
|
BACKEND_VERSION = "0.7.7"
|
||||||
@@ -25,5 +25,15 @@ services:
|
|||||||
postgres:
|
postgres:
|
||||||
condition: service_healthy
|
condition: service_healthy
|
||||||
|
|
||||||
|
worker:
|
||||||
|
build: .
|
||||||
|
command: python -m app.worker
|
||||||
|
env_file: .env
|
||||||
|
environment:
|
||||||
|
DATABASE_URL: postgresql+psycopg://${POSTGRES_USER:-miem}:${POSTGRES_PASSWORD:-miem_password}@postgres:5432/${POSTGRES_DB:-miem_workers}
|
||||||
|
depends_on:
|
||||||
|
postgres:
|
||||||
|
condition: service_healthy
|
||||||
|
|
||||||
volumes:
|
volumes:
|
||||||
postgres_data:
|
postgres_data:
|
||||||
+2
-2
@@ -1,7 +1,7 @@
|
|||||||
[project]
|
[project]
|
||||||
name = "miem-workers"
|
name = "miem-workers"
|
||||||
version = "0.7.5"
|
version = "0.7.7"
|
||||||
description = "MIEM employees parser, admin API, and MCP server"
|
description = "MIEM employees parser, admin API, and web admin"
|
||||||
requires-python = ">=3.11"
|
requires-python = ">=3.11"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"apscheduler>=3.10.4",
|
"apscheduler>=3.10.4",
|
||||||
|
|||||||
@@ -0,0 +1,130 @@
|
|||||||
|
from datetime import datetime, timezone
|
||||||
|
from types import SimpleNamespace
|
||||||
|
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
from sqlalchemy import create_engine, select
|
||||||
|
from sqlalchemy.orm import sessionmaker
|
||||||
|
from sqlalchemy.pool import StaticPool
|
||||||
|
|
||||||
|
from app.config import Settings, get_settings
|
||||||
|
from app.db import Base, get_db
|
||||||
|
from app.main import app
|
||||||
|
from app.models import CrawlRun, CrawlRunEmployeeChange, Employee
|
||||||
|
from app.security import SESSION_COOKIE, sign_session
|
||||||
|
|
||||||
|
|
||||||
|
def test_health_returns_versions():
|
||||||
|
response = TestClient(app).get("/api/health")
|
||||||
|
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert response.json()["backend_version"] == "0.7.7"
|
||||||
|
|
||||||
|
|
||||||
|
def test_mcp_endpoint_is_removed():
|
||||||
|
assert TestClient(app).get("/mcp").status_code == 404
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_employees_and_stats_require_admin_session():
|
||||||
|
engine = create_engine("sqlite:///:memory:", connect_args={"check_same_thread": False}, poolclass=StaticPool)
|
||||||
|
Base.metadata.create_all(engine)
|
||||||
|
session_factory = sessionmaker(bind=engine)
|
||||||
|
db = session_factory()
|
||||||
|
employee = Employee(
|
||||||
|
profile_key="staff:alpha",
|
||||||
|
profile_type="staff",
|
||||||
|
profile_id="alpha",
|
||||||
|
canonical_url="https://www.hse.ru/staff/alpha",
|
||||||
|
full_name="Alpha Person",
|
||||||
|
status="active",
|
||||||
|
first_seen_at=datetime.now(timezone.utc),
|
||||||
|
last_seen_at=datetime.now(timezone.utc),
|
||||||
|
current_data={"contacts": {"emails": ["alpha@hse.ru"]}, "sections": []},
|
||||||
|
)
|
||||||
|
run = CrawlRun(source_url="https://miem.hse.ru/persons", status="completed", new_count=1)
|
||||||
|
db.add_all([employee, run])
|
||||||
|
db.commit()
|
||||||
|
db.add(CrawlRunEmployeeChange(
|
||||||
|
crawl_run_id=run.id,
|
||||||
|
employee_id=employee.id,
|
||||||
|
profile_key=employee.profile_key,
|
||||||
|
profile_url=employee.canonical_url,
|
||||||
|
full_name=employee.full_name,
|
||||||
|
change_type="new",
|
||||||
|
profile_available=True,
|
||||||
|
message="added",
|
||||||
|
))
|
||||||
|
db.commit()
|
||||||
|
run_id = run.id
|
||||||
|
db.close()
|
||||||
|
|
||||||
|
settings = Settings(admin_username="admin", admin_password="password", session_secret="session-secret")
|
||||||
|
|
||||||
|
def override_db():
|
||||||
|
session = session_factory()
|
||||||
|
try:
|
||||||
|
yield session
|
||||||
|
finally:
|
||||||
|
session.close()
|
||||||
|
|
||||||
|
app.dependency_overrides[get_db] = override_db
|
||||||
|
app.dependency_overrides[get_settings] = lambda: settings
|
||||||
|
client = TestClient(app)
|
||||||
|
client.cookies.set(SESSION_COOKIE, sign_session("admin", settings))
|
||||||
|
|
||||||
|
employees = client.get("/api/employees", params={"q": "Alpha", "has_email": True})
|
||||||
|
stats = client.get("/api/stats")
|
||||||
|
run_details = client.get(f"/api/crawl-runs/{run_id}")
|
||||||
|
|
||||||
|
assert employees.status_code == 200
|
||||||
|
assert employees.json()["total"] == 1
|
||||||
|
assert stats.status_code == 200
|
||||||
|
assert stats.json()["new_in_last_run"] == 1
|
||||||
|
assert run_details.status_code == 200
|
||||||
|
assert run_details.json()["changes"]["new"][0]["full_name"] == "Alpha Person"
|
||||||
|
app.dependency_overrides.clear()
|
||||||
|
|
||||||
|
|
||||||
|
def test_admin_refresh_employee_route_updates_only_requested_employee(monkeypatch):
|
||||||
|
engine = create_engine("sqlite:///:memory:", connect_args={"check_same_thread": False}, poolclass=StaticPool)
|
||||||
|
Base.metadata.create_all(engine)
|
||||||
|
session_factory = sessionmaker(bind=engine)
|
||||||
|
db = session_factory()
|
||||||
|
db.add(Employee(
|
||||||
|
profile_key="org_person:133709486",
|
||||||
|
profile_type="org_person",
|
||||||
|
profile_id="133709486",
|
||||||
|
canonical_url="https://www.hse.ru/org/persons/133709486",
|
||||||
|
full_name="Будков Юрий Алексеевич",
|
||||||
|
status="active",
|
||||||
|
))
|
||||||
|
db.commit()
|
||||||
|
employee_id = db.scalar(select(Employee.id))
|
||||||
|
db.close()
|
||||||
|
|
||||||
|
settings = Settings(admin_username="admin", admin_password="password", session_secret="session-secret")
|
||||||
|
|
||||||
|
def override_db():
|
||||||
|
session = session_factory()
|
||||||
|
try:
|
||||||
|
yield session
|
||||||
|
finally:
|
||||||
|
session.close()
|
||||||
|
|
||||||
|
calls = []
|
||||||
|
|
||||||
|
def fake_refresh_employee(db, refreshed_employee, route_settings):
|
||||||
|
calls.append((refreshed_employee.id, route_settings))
|
||||||
|
return SimpleNamespace(status="completed")
|
||||||
|
|
||||||
|
app.dependency_overrides[get_db] = override_db
|
||||||
|
app.dependency_overrides[get_settings] = lambda: settings
|
||||||
|
monkeypatch.setattr("app.admin.refresh_employee", fake_refresh_employee)
|
||||||
|
client = TestClient(app)
|
||||||
|
client.cookies.set(SESSION_COOKIE, sign_session("admin", settings))
|
||||||
|
|
||||||
|
response = client.post(f"/admin/employees/{employee_id}/refresh", follow_redirects=False)
|
||||||
|
|
||||||
|
assert response.status_code == 303
|
||||||
|
assert response.headers["location"] == f"/admin/employees/{employee_id}?refresh_status=success"
|
||||||
|
assert calls == [(employee_id, settings)]
|
||||||
|
app.dependency_overrides.clear()
|
||||||
@@ -1,532 +0,0 @@
|
|||||||
import json
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
from types import SimpleNamespace
|
|
||||||
|
|
||||||
from fastapi.testclient import TestClient
|
|
||||||
from sqlalchemy import create_engine, select
|
|
||||||
from sqlalchemy.orm import sessionmaker
|
|
||||||
from sqlalchemy.pool import StaticPool
|
|
||||||
|
|
||||||
from app.config import Settings, get_settings
|
|
||||||
from app.db import Base, get_db
|
|
||||||
from app.main import app
|
|
||||||
from app.models import CrawlRun, CrawlRunEmployeeChange, Employee, EmployeePublication
|
|
||||||
from app.security import SESSION_COOKIE, sign_session
|
|
||||||
|
|
||||||
|
|
||||||
def test_health_returns_versions():
|
|
||||||
client = TestClient(app)
|
|
||||||
|
|
||||||
response = client.get("/api/health")
|
|
||||||
|
|
||||||
assert response.status_code == 200
|
|
||||||
assert response.json()["backend_version"] == "0.7.3"
|
|
||||||
|
|
||||||
|
|
||||||
def test_mcp_lists_tools_without_auth_and_ignores_auth_header():
|
|
||||||
engine = create_engine(
|
|
||||||
"sqlite:///:memory:",
|
|
||||||
connect_args={"check_same_thread": False},
|
|
||||||
poolclass=StaticPool,
|
|
||||||
)
|
|
||||||
Base.metadata.create_all(engine)
|
|
||||||
Session = sessionmaker(bind=engine)
|
|
||||||
|
|
||||||
def override_db():
|
|
||||||
session = Session()
|
|
||||||
try:
|
|
||||||
yield session
|
|
||||||
finally:
|
|
||||||
session.close()
|
|
||||||
|
|
||||||
app.dependency_overrides[get_db] = override_db
|
|
||||||
client = TestClient(app)
|
|
||||||
|
|
||||||
without_auth = client.post("/mcp", json={"jsonrpc": "2.0", "id": 1, "method": "tools/list", "params": {}})
|
|
||||||
with_auth = client.post(
|
|
||||||
"/mcp",
|
|
||||||
headers={"Authorization": "Bearer anything"},
|
|
||||||
json={"jsonrpc": "2.0", "id": 1, "method": "tools/list", "params": {}},
|
|
||||||
)
|
|
||||||
|
|
||||||
assert without_auth.status_code == 200
|
|
||||||
assert with_auth.status_code == 200
|
|
||||||
tool_names = {tool["name"] for tool in without_auth.json()["result"]["tools"]}
|
|
||||||
assert "search_employees" in tool_names
|
|
||||||
assert "get_service_info" in tool_names
|
|
||||||
assert "sync_employees" in tool_names
|
|
||||||
assert any(tool["name"] == "get_crawl_run_details" for tool in without_auth.json()["result"]["tools"])
|
|
||||||
assert with_auth.json()["result"]["tools"] == without_auth.json()["result"]["tools"]
|
|
||||||
|
|
||||||
app.dependency_overrides.clear()
|
|
||||||
|
|
||||||
|
|
||||||
def test_mcp_search_employees_returns_matching_employee():
|
|
||||||
engine = create_engine(
|
|
||||||
"sqlite:///:memory:",
|
|
||||||
connect_args={"check_same_thread": False},
|
|
||||||
poolclass=StaticPool,
|
|
||||||
)
|
|
||||||
Base.metadata.create_all(engine)
|
|
||||||
Session = sessionmaker(bind=engine)
|
|
||||||
session = Session()
|
|
||||||
session.add(
|
|
||||||
Employee(
|
|
||||||
profile_key="staff:avsergeev",
|
|
||||||
profile_type="staff",
|
|
||||||
profile_id="avsergeev",
|
|
||||||
canonical_url="https://www.hse.ru/staff/avsergeev",
|
|
||||||
full_name="Сергеев Алексей Викторович",
|
|
||||||
status="active",
|
|
||||||
first_seen_at=datetime.now(timezone.utc),
|
|
||||||
last_seen_at=datetime.now(timezone.utc),
|
|
||||||
current_data={"sections": []},
|
|
||||||
)
|
|
||||||
)
|
|
||||||
session.commit()
|
|
||||||
session.close()
|
|
||||||
|
|
||||||
def override_db():
|
|
||||||
db = Session()
|
|
||||||
try:
|
|
||||||
yield db
|
|
||||||
finally:
|
|
||||||
db.close()
|
|
||||||
|
|
||||||
app.dependency_overrides[get_db] = override_db
|
|
||||||
client = TestClient(app)
|
|
||||||
|
|
||||||
response = client.post(
|
|
||||||
"/mcp",
|
|
||||||
json={
|
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"id": 1,
|
|
||||||
"method": "tools/call",
|
|
||||||
"params": {"name": "search_employees", "arguments": {"query": "Сергеев"}},
|
|
||||||
},
|
|
||||||
)
|
|
||||||
|
|
||||||
assert response.status_code == 200
|
|
||||||
assert "Сергеев Алексей Викторович" in response.json()["result"]["content"][0]["text"]
|
|
||||||
|
|
||||||
app.dependency_overrides.clear()
|
|
||||||
|
|
||||||
|
|
||||||
def test_mcp_service_info_returns_tools_and_dataset_hash():
|
|
||||||
engine = create_engine(
|
|
||||||
"sqlite:///:memory:",
|
|
||||||
connect_args={"check_same_thread": False},
|
|
||||||
poolclass=StaticPool,
|
|
||||||
)
|
|
||||||
Base.metadata.create_all(engine)
|
|
||||||
Session = sessionmaker(bind=engine)
|
|
||||||
session = Session()
|
|
||||||
session.add(
|
|
||||||
Employee(
|
|
||||||
profile_key="staff:alpha",
|
|
||||||
profile_type="staff",
|
|
||||||
profile_id="alpha",
|
|
||||||
canonical_url="https://www.hse.ru/staff/alpha",
|
|
||||||
full_name="Alpha Person",
|
|
||||||
status="active",
|
|
||||||
current_checksum="a" * 64,
|
|
||||||
current_data={"sections": []},
|
|
||||||
)
|
|
||||||
)
|
|
||||||
session.commit()
|
|
||||||
session.close()
|
|
||||||
|
|
||||||
def override_db():
|
|
||||||
db = Session()
|
|
||||||
try:
|
|
||||||
yield db
|
|
||||||
finally:
|
|
||||||
db.close()
|
|
||||||
|
|
||||||
app.dependency_overrides[get_db] = override_db
|
|
||||||
client = TestClient(app)
|
|
||||||
|
|
||||||
response = client.post(
|
|
||||||
"/mcp",
|
|
||||||
json={"jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": {"name": "get_service_info", "arguments": {}}},
|
|
||||||
)
|
|
||||||
|
|
||||||
assert response.status_code == 200
|
|
||||||
payload = json.loads(response.json()["result"]["content"][0]["text"])
|
|
||||||
assert payload["service_name"] == "miem-employees"
|
|
||||||
assert payload["backend_version"] == "0.7.3"
|
|
||||||
assert payload["dataset"]["hash"]
|
|
||||||
assert any(tool["name"] == "sync_employees" for tool in payload["tools"])
|
|
||||||
|
|
||||||
app.dependency_overrides.clear()
|
|
||||||
|
|
||||||
|
|
||||||
def test_mcp_list_employee_publications_prefers_stored_publications_with_fallback():
|
|
||||||
engine = create_engine(
|
|
||||||
"sqlite:///:memory:",
|
|
||||||
connect_args={"check_same_thread": False},
|
|
||||||
poolclass=StaticPool,
|
|
||||||
)
|
|
||||||
Base.metadata.create_all(engine)
|
|
||||||
Session = sessionmaker(bind=engine)
|
|
||||||
session = Session()
|
|
||||||
stored_employee = Employee(
|
|
||||||
profile_key="staff:stored",
|
|
||||||
profile_type="staff",
|
|
||||||
profile_id="stored",
|
|
||||||
canonical_url="https://www.hse.ru/staff/stored",
|
|
||||||
full_name="Stored Person",
|
|
||||||
status="active",
|
|
||||||
current_data={
|
|
||||||
"sections": [
|
|
||||||
{
|
|
||||||
"type": "publications",
|
|
||||||
"publications": [{"title": "Old JSON Publication", "url": "https://example.test/old"}],
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
)
|
|
||||||
fallback_employee = Employee(
|
|
||||||
profile_key="staff:fallback",
|
|
||||||
profile_type="staff",
|
|
||||||
profile_id="fallback",
|
|
||||||
canonical_url="https://www.hse.ru/staff/fallback",
|
|
||||||
full_name="Fallback Person",
|
|
||||||
status="active",
|
|
||||||
current_data={
|
|
||||||
"sections": [
|
|
||||||
{
|
|
||||||
"type": "publications",
|
|
||||||
"publications": [{"title": "Fallback Publication", "url": "https://example.test/fallback"}],
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
)
|
|
||||||
session.add_all([stored_employee, fallback_employee])
|
|
||||||
session.commit()
|
|
||||||
session.add(
|
|
||||||
EmployeePublication(
|
|
||||||
employee_id=stored_employee.id,
|
|
||||||
publication_id="pub-1",
|
|
||||||
title="Stored Publication",
|
|
||||||
year=2024,
|
|
||||||
publication_type="ARTICLE",
|
|
||||||
url="https://publications.hse.ru/view/pub-1",
|
|
||||||
doi_url="https://doi.org/10.1/test",
|
|
||||||
citation_text="Stored Citation",
|
|
||||||
annotation={"ru": "Аннотация", "en": "Abstract"},
|
|
||||||
description={"main": "Stored Citation"},
|
|
||||||
authors=[{"id": "1", "title_ru": "Автор", "is_current_employee": True}],
|
|
||||||
source_hash="a" * 64,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
session.commit()
|
|
||||||
session.close()
|
|
||||||
|
|
||||||
def override_db():
|
|
||||||
db = Session()
|
|
||||||
try:
|
|
||||||
yield db
|
|
||||||
finally:
|
|
||||||
db.close()
|
|
||||||
|
|
||||||
app.dependency_overrides[get_db] = override_db
|
|
||||||
client = TestClient(app)
|
|
||||||
|
|
||||||
stored_response = client.post(
|
|
||||||
"/mcp",
|
|
||||||
json={
|
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"id": 1,
|
|
||||||
"method": "tools/call",
|
|
||||||
"params": {"name": "list_employee_publications", "arguments": {"profile_id_or_url": "stored"}},
|
|
||||||
},
|
|
||||||
)
|
|
||||||
fallback_response = client.post(
|
|
||||||
"/mcp",
|
|
||||||
json={
|
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"id": 2,
|
|
||||||
"method": "tools/call",
|
|
||||||
"params": {"name": "list_employee_publications", "arguments": {"profile_id_or_url": "fallback"}},
|
|
||||||
},
|
|
||||||
)
|
|
||||||
|
|
||||||
stored_payload = json.loads(stored_response.json()["result"]["content"][0]["text"])
|
|
||||||
fallback_payload = json.loads(fallback_response.json()["result"]["content"][0]["text"])
|
|
||||||
assert stored_payload["items"][0]["title"] == "Stored Publication"
|
|
||||||
assert stored_payload["items"][0]["doi_url"] == "https://doi.org/10.1/test"
|
|
||||||
assert stored_payload["items"][0]["annotation"] == {"ru": "Аннотация", "en": "Abstract"}
|
|
||||||
assert stored_payload["items"][0]["authors"] == [{"id": "1", "title_ru": "Автор", "is_current_employee": True}]
|
|
||||||
assert fallback_payload["items"][0]["title"] == "Fallback Publication"
|
|
||||||
|
|
||||||
app.dependency_overrides.clear()
|
|
||||||
|
|
||||||
|
|
||||||
def test_mcp_sync_employees_full_empty_and_unknown_hash_modes():
|
|
||||||
engine = create_engine(
|
|
||||||
"sqlite:///:memory:",
|
|
||||||
connect_args={"check_same_thread": False},
|
|
||||||
poolclass=StaticPool,
|
|
||||||
)
|
|
||||||
Base.metadata.create_all(engine)
|
|
||||||
Session = sessionmaker(bind=engine)
|
|
||||||
session = Session()
|
|
||||||
session.add(
|
|
||||||
Employee(
|
|
||||||
profile_key="staff:alpha",
|
|
||||||
profile_type="staff",
|
|
||||||
profile_id="alpha",
|
|
||||||
canonical_url="https://www.hse.ru/staff/alpha",
|
|
||||||
full_name="Alpha Person",
|
|
||||||
status="active",
|
|
||||||
current_checksum="a" * 64,
|
|
||||||
current_data={"sections": [{"type": "paragraphs"}]},
|
|
||||||
)
|
|
||||||
)
|
|
||||||
session.commit()
|
|
||||||
session.close()
|
|
||||||
|
|
||||||
def override_db():
|
|
||||||
db = Session()
|
|
||||||
try:
|
|
||||||
yield db
|
|
||||||
finally:
|
|
||||||
db.close()
|
|
||||||
|
|
||||||
app.dependency_overrides[get_db] = override_db
|
|
||||||
client = TestClient(app)
|
|
||||||
|
|
||||||
full_response = client.post(
|
|
||||||
"/mcp",
|
|
||||||
json={"jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": {"name": "sync_employees", "arguments": {}}},
|
|
||||||
)
|
|
||||||
full_payload = json.loads(full_response.json()["result"]["content"][0]["text"])
|
|
||||||
current_hash = full_payload["to_hash"]
|
|
||||||
|
|
||||||
empty_response = client.post(
|
|
||||||
"/mcp",
|
|
||||||
json={
|
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"id": 2,
|
|
||||||
"method": "tools/call",
|
|
||||||
"params": {"name": "sync_employees", "arguments": {"client_hash": current_hash}},
|
|
||||||
},
|
|
||||||
)
|
|
||||||
empty_payload = json.loads(empty_response.json()["result"]["content"][0]["text"])
|
|
||||||
|
|
||||||
unknown_response = client.post(
|
|
||||||
"/mcp",
|
|
||||||
json={
|
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"id": 3,
|
|
||||||
"method": "tools/call",
|
|
||||||
"params": {"name": "sync_employees", "arguments": {"client_hash": "missing"}},
|
|
||||||
},
|
|
||||||
)
|
|
||||||
unknown_payload = json.loads(unknown_response.json()["result"]["content"][0]["text"])
|
|
||||||
|
|
||||||
assert full_payload["mode"] == "full"
|
|
||||||
assert full_payload["items"][0]["data"] == {"sections": [{"type": "paragraphs"}]}
|
|
||||||
assert empty_payload["mode"] == "delta"
|
|
||||||
assert empty_payload["changes"] == {"added": [], "updated": [], "dismissed": [], "removed": []}
|
|
||||||
assert unknown_payload["mode"] == "full"
|
|
||||||
assert unknown_payload["reason"] == "unknown_client_hash"
|
|
||||||
|
|
||||||
app.dependency_overrides.clear()
|
|
||||||
|
|
||||||
|
|
||||||
def test_mcp_get_crawl_run_details_returns_changes():
|
|
||||||
engine = create_engine(
|
|
||||||
"sqlite:///:memory:",
|
|
||||||
connect_args={"check_same_thread": False},
|
|
||||||
poolclass=StaticPool,
|
|
||||||
)
|
|
||||||
Base.metadata.create_all(engine)
|
|
||||||
Session = sessionmaker(bind=engine)
|
|
||||||
session = Session()
|
|
||||||
run = CrawlRun(source_url="https://miem.hse.ru/persons", status="completed", new_count=1)
|
|
||||||
employee = Employee(
|
|
||||||
profile_key="staff:new",
|
|
||||||
profile_type="staff",
|
|
||||||
profile_id="new",
|
|
||||||
canonical_url="https://www.hse.ru/staff/new",
|
|
||||||
full_name="New Person",
|
|
||||||
status="active",
|
|
||||||
first_seen_at=datetime.now(timezone.utc),
|
|
||||||
last_seen_at=datetime.now(timezone.utc),
|
|
||||||
)
|
|
||||||
session.add_all([run, employee])
|
|
||||||
session.commit()
|
|
||||||
session.add(
|
|
||||||
CrawlRunEmployeeChange(
|
|
||||||
crawl_run_id=run.id,
|
|
||||||
employee_id=employee.id,
|
|
||||||
profile_key=employee.profile_key,
|
|
||||||
profile_url=employee.canonical_url,
|
|
||||||
full_name=employee.full_name,
|
|
||||||
change_type="new",
|
|
||||||
profile_available=True,
|
|
||||||
message="added",
|
|
||||||
)
|
|
||||||
)
|
|
||||||
session.commit()
|
|
||||||
run_id = run.id
|
|
||||||
session.close()
|
|
||||||
|
|
||||||
def override_db():
|
|
||||||
db = Session()
|
|
||||||
try:
|
|
||||||
yield db
|
|
||||||
finally:
|
|
||||||
db.close()
|
|
||||||
|
|
||||||
app.dependency_overrides[get_db] = override_db
|
|
||||||
client = TestClient(app)
|
|
||||||
|
|
||||||
response = client.post(
|
|
||||||
"/mcp",
|
|
||||||
json={
|
|
||||||
"jsonrpc": "2.0",
|
|
||||||
"id": 1,
|
|
||||||
"method": "tools/call",
|
|
||||||
"params": {"name": "get_crawl_run_details", "arguments": {"run_id": run_id}},
|
|
||||||
},
|
|
||||||
)
|
|
||||||
|
|
||||||
assert response.status_code == 200
|
|
||||||
text = response.json()["result"]["content"][0]["text"]
|
|
||||||
assert "New Person" in text
|
|
||||||
assert "changes_detail_available" in text
|
|
||||||
|
|
||||||
app.dependency_overrides.clear()
|
|
||||||
|
|
||||||
|
|
||||||
def test_mcp_protected_resource_metadata_route_is_removed():
|
|
||||||
client = TestClient(app)
|
|
||||||
|
|
||||||
response = client.get("/.well-known/oauth-protected-resource")
|
|
||||||
|
|
||||||
assert response.status_code == 404
|
|
||||||
|
|
||||||
|
|
||||||
def test_api_employees_and_stats_require_admin_session():
|
|
||||||
engine = create_engine(
|
|
||||||
"sqlite:///:memory:",
|
|
||||||
connect_args={"check_same_thread": False},
|
|
||||||
poolclass=StaticPool,
|
|
||||||
)
|
|
||||||
Base.metadata.create_all(engine)
|
|
||||||
Session = sessionmaker(bind=engine)
|
|
||||||
db = Session()
|
|
||||||
db.add(
|
|
||||||
Employee(
|
|
||||||
profile_key="staff:alpha",
|
|
||||||
profile_type="staff",
|
|
||||||
profile_id="alpha",
|
|
||||||
canonical_url="https://www.hse.ru/staff/alpha",
|
|
||||||
full_name="Alpha Person",
|
|
||||||
status="active",
|
|
||||||
first_seen_at=datetime.now(timezone.utc),
|
|
||||||
last_seen_at=datetime.now(timezone.utc),
|
|
||||||
current_data={"contacts": {"emails": ["alpha@hse.ru"]}, "sections": []},
|
|
||||||
)
|
|
||||||
)
|
|
||||||
run = CrawlRun(source_url="https://miem.hse.ru/persons", status="completed", new_count=1)
|
|
||||||
db.add(run)
|
|
||||||
db.commit()
|
|
||||||
db.add(
|
|
||||||
CrawlRunEmployeeChange(
|
|
||||||
crawl_run_id=run.id,
|
|
||||||
employee_id=1,
|
|
||||||
profile_key="staff:alpha",
|
|
||||||
profile_url="https://www.hse.ru/staff/alpha",
|
|
||||||
full_name="Alpha Person",
|
|
||||||
change_type="new",
|
|
||||||
profile_available=True,
|
|
||||||
message="added",
|
|
||||||
)
|
|
||||||
)
|
|
||||||
db.commit()
|
|
||||||
run_id = run.id
|
|
||||||
db.close()
|
|
||||||
|
|
||||||
settings = Settings(admin_username="admin", admin_password="password", session_secret="session-secret")
|
|
||||||
|
|
||||||
def override_db():
|
|
||||||
session = Session()
|
|
||||||
try:
|
|
||||||
yield session
|
|
||||||
finally:
|
|
||||||
session.close()
|
|
||||||
|
|
||||||
app.dependency_overrides[get_db] = override_db
|
|
||||||
app.dependency_overrides[get_settings] = lambda: settings
|
|
||||||
client = TestClient(app)
|
|
||||||
client.cookies.set(SESSION_COOKIE, sign_session("admin", settings))
|
|
||||||
|
|
||||||
employees = client.get("/api/employees", params={"q": "Alpha", "has_email": True})
|
|
||||||
stats = client.get("/api/stats")
|
|
||||||
run_details = client.get(f"/api/crawl-runs/{run_id}")
|
|
||||||
|
|
||||||
assert employees.status_code == 200
|
|
||||||
assert employees.json()["total"] == 1
|
|
||||||
assert stats.status_code == 200
|
|
||||||
assert stats.json()["new_in_last_run"] == 1
|
|
||||||
assert run_details.status_code == 200
|
|
||||||
assert run_details.json()["changes"]["new"][0]["full_name"] == "Alpha Person"
|
|
||||||
|
|
||||||
app.dependency_overrides.clear()
|
|
||||||
|
|
||||||
|
|
||||||
def test_admin_refresh_employee_route_updates_only_requested_employee(monkeypatch):
|
|
||||||
engine = create_engine(
|
|
||||||
"sqlite:///:memory:",
|
|
||||||
connect_args={"check_same_thread": False},
|
|
||||||
poolclass=StaticPool,
|
|
||||||
)
|
|
||||||
Base.metadata.create_all(engine)
|
|
||||||
Session = sessionmaker(bind=engine)
|
|
||||||
db = Session()
|
|
||||||
db.add(
|
|
||||||
Employee(
|
|
||||||
profile_key="org_person:133709486",
|
|
||||||
profile_type="org_person",
|
|
||||||
profile_id="133709486",
|
|
||||||
canonical_url="https://www.hse.ru/org/persons/133709486",
|
|
||||||
full_name="Будков Юрий Алексеевич",
|
|
||||||
status="active",
|
|
||||||
)
|
|
||||||
)
|
|
||||||
db.commit()
|
|
||||||
employee_id = db.scalar(select(Employee.id))
|
|
||||||
db.close()
|
|
||||||
|
|
||||||
settings = Settings(admin_username="admin", admin_password="password", session_secret="session-secret")
|
|
||||||
|
|
||||||
def override_db():
|
|
||||||
session = Session()
|
|
||||||
try:
|
|
||||||
yield session
|
|
||||||
finally:
|
|
||||||
session.close()
|
|
||||||
|
|
||||||
calls = []
|
|
||||||
|
|
||||||
def fake_refresh_employee(db, refreshed_employee, route_settings):
|
|
||||||
calls.append((refreshed_employee.id, route_settings))
|
|
||||||
return SimpleNamespace(status="completed")
|
|
||||||
|
|
||||||
app.dependency_overrides[get_db] = override_db
|
|
||||||
app.dependency_overrides[get_settings] = lambda: settings
|
|
||||||
monkeypatch.setattr("app.admin.refresh_employee", fake_refresh_employee)
|
|
||||||
client = TestClient(app)
|
|
||||||
client.cookies.set(SESSION_COOKIE, sign_session("admin", settings))
|
|
||||||
|
|
||||||
response = client.post(f"/admin/employees/{employee_id}/refresh", follow_redirects=False)
|
|
||||||
|
|
||||||
assert response.status_code == 303
|
|
||||||
assert response.headers["location"] == f"/admin/employees/{employee_id}?refresh_status=success"
|
|
||||||
assert calls == [(employee_id, settings)]
|
|
||||||
|
|
||||||
app.dependency_overrides.clear()
|
|
||||||
@@ -1,88 +0,0 @@
|
|||||||
from datetime import datetime, timezone
|
|
||||||
|
|
||||||
from app.models import Employee
|
|
||||||
from app.services.dataset_versions import get_or_create_current_version, sync_employees_payload
|
|
||||||
|
|
||||||
|
|
||||||
def _employee(profile_key: str, checksum: str, *, status: str = "active") -> Employee:
|
|
||||||
return Employee(
|
|
||||||
profile_key=profile_key,
|
|
||||||
profile_type=profile_key.split(":", 1)[0],
|
|
||||||
profile_id=profile_key.split(":", 1)[1],
|
|
||||||
canonical_url=f"https://www.hse.ru/{profile_key}",
|
|
||||||
full_name=profile_key,
|
|
||||||
status=status,
|
|
||||||
first_seen_at=datetime.now(timezone.utc),
|
|
||||||
last_seen_at=datetime.now(timezone.utc),
|
|
||||||
current_data={"profile_key": profile_key},
|
|
||||||
current_checksum=checksum,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def test_dataset_version_hash_is_stable_for_same_employee_state(db_session):
|
|
||||||
db_session.add(_employee("staff:alpha", "a" * 64))
|
|
||||||
db_session.commit()
|
|
||||||
|
|
||||||
first = get_or_create_current_version(db_session)
|
|
||||||
db_session.commit()
|
|
||||||
second = get_or_create_current_version(db_session)
|
|
||||||
|
|
||||||
assert second.id == first.id
|
|
||||||
assert second.hash == first.hash
|
|
||||||
assert second.employee_count == 1
|
|
||||||
|
|
||||||
|
|
||||||
def test_dataset_version_hash_changes_when_employee_checksum_changes(db_session):
|
|
||||||
employee = _employee("staff:alpha", "a" * 64)
|
|
||||||
db_session.add(employee)
|
|
||||||
db_session.commit()
|
|
||||||
first = get_or_create_current_version(db_session)
|
|
||||||
db_session.commit()
|
|
||||||
|
|
||||||
employee.current_checksum = "b" * 64
|
|
||||||
db_session.commit()
|
|
||||||
second = get_or_create_current_version(db_session)
|
|
||||||
|
|
||||||
assert second.hash != first.hash
|
|
||||||
assert second.previous_hash == first.hash
|
|
||||||
|
|
||||||
|
|
||||||
def test_sync_employees_diff_spans_multiple_intermediate_versions(db_session):
|
|
||||||
alpha = _employee("staff:alpha", "a" * 64)
|
|
||||||
db_session.add(alpha)
|
|
||||||
db_session.commit()
|
|
||||||
first = get_or_create_current_version(db_session)
|
|
||||||
db_session.commit()
|
|
||||||
|
|
||||||
beta = _employee("staff:beta", "b" * 64)
|
|
||||||
db_session.add(beta)
|
|
||||||
db_session.commit()
|
|
||||||
get_or_create_current_version(db_session)
|
|
||||||
db_session.commit()
|
|
||||||
|
|
||||||
alpha.current_checksum = "c" * 64
|
|
||||||
alpha.current_data = {"profile_key": "staff:alpha", "changed": True}
|
|
||||||
db_session.commit()
|
|
||||||
|
|
||||||
payload = sync_employees_payload(db_session, client_hash=first.hash, include_data=False)
|
|
||||||
|
|
||||||
assert payload["mode"] == "delta"
|
|
||||||
assert [item["profile_key"] for item in payload["changes"]["added"]] == ["staff:beta"]
|
|
||||||
assert [item["profile_key"] for item in payload["changes"]["updated"]] == ["staff:alpha"]
|
|
||||||
assert payload["changes"]["dismissed"] == []
|
|
||||||
assert payload["changes"]["removed"] == []
|
|
||||||
|
|
||||||
|
|
||||||
def test_sync_employees_reports_dismissed_as_tombstone(db_session):
|
|
||||||
alpha = _employee("staff:alpha", "a" * 64)
|
|
||||||
db_session.add(alpha)
|
|
||||||
db_session.commit()
|
|
||||||
first = get_or_create_current_version(db_session)
|
|
||||||
db_session.commit()
|
|
||||||
|
|
||||||
alpha.status = "dismissed"
|
|
||||||
db_session.commit()
|
|
||||||
payload = sync_employees_payload(db_session, client_hash=first.hash, include_data=False)
|
|
||||||
|
|
||||||
assert payload["changes"]["dismissed"][0]["profile_key"] == "staff:alpha"
|
|
||||||
assert payload["changes"]["dismissed"][0]["status"] == "dismissed"
|
|
||||||
Reference in new issue
Block a user