Compare commits

...

19 Commits

Author SHA1 Message Date
Anton
deaecd8d3b feat: make dashboard metrics and run rows clickable 2026-05-07 09:35:44 +03:00
e4d4271e32 Merge pull request 'feat: track crawl run employee changes and verify dismissals' (#14) from feature/crawl-run-change-details into main
Reviewed-on: #14
2026-05-06 12:14:51 +00:00
Anton
d0459a2c30 feat: track crawl run employee changes and verify dismissals 2026-05-06 15:13:15 +03:00
Anton
2331c7a28d chore: removes sensitive data from docker file 2026-04-29 16:16:06 +03:00
064c34ea32 Merge pull request 'feat: adds oauth server to docker' (#13) from feature/add-oauth-server into main
Reviewed-on: #13
2026-04-29 12:59:55 +00:00
Anton
6a98ae4246 feat: adds oauth server to docker 2026-04-29 15:59:18 +03:00
a6f2883091 Merge pull request 'feat: requires OAuth-only auth mode for MCP agents' (#12) from feature/mcp-oauth-oidc into main
Reviewed-on: #12
2026-04-29 12:22:25 +00:00
Anton
d20b4f396b feat: requires OAuth-only auth mode for MCP agents 2026-04-29 15:08:18 +03:00
c7027bb503 Merge pull request 'feat: adds OAuth/OIDC authentication for MCP' (#11) from feature/mcp-oauth-oidc into main
Reviewed-on: #11
2026-04-29 11:35:00 +00:00
Anton
ad0b15cc6e feat: adds OAuth/OIDC authentication for MCP 2026-04-29 14:33:29 +03:00
af864ecb44 Merge pull request 'fix: enrich HSE profile parsing with publications and theses' (#10) from fix/hse-profile-parser-publications-vkr-pagination into main
Reviewed-on: #10
2026-04-29 11:16:17 +00:00
Anton
cc9481fc6c fix: enrich HSE profile parsing with publications and theses 2026-04-29 14:15:29 +03:00
cf578ce699 Merge pull request 'fix: allow empty CRAWL_LIMIT env value' (#9) from fix/empty-crawl-limit-env into main
Reviewed-on: #9
2026-04-29 09:50:34 +00:00
Anton
765efa1a1c fix: allow empty CRAWL_LIMIT env value 2026-04-29 12:49:58 +03:00
86330885e3 Merge pull request 'fix: localize admin UI and simplify employees navigation' (#8) from fix/admin-russian-ux-cleanup into main
Reviewed-on: #8
2026-04-29 09:39:42 +00:00
Anton
866e2b44d5 fix: localize admin UI and simplify employees navigation 2026-04-29 12:39:16 +03:00
f411de740e Merge pull request 'fix: avoid Jinja dict method collisions in admin templates' (#7) from fix/jinja-dict-method-collisions into main
Reviewed-on: #7
2026-04-29 09:12:13 +00:00
Anton
cdfbb26875 fix: avoid Jinja dict method collisions in admin templates 2026-04-29 12:11:16 +03:00
5eaad38076 Merge pull request 'fix: avoid Jinja dict items collision in employee card' (#6) from fix/employee-card-contact-items into main
Reviewed-on: #6
2026-04-29 08:35:13 +00:00
35 changed files with 1576 additions and 191 deletions

View File

@@ -15,6 +15,12 @@ ADMIN_USERNAME=admin
ADMIN_PASSWORD=change-me ADMIN_PASSWORD=change-me
SESSION_SECRET=change-me-session-secret SESSION_SECRET=change-me-session-secret
MCP_TOKEN=change-me-mcp-token MCP_TOKEN=change-me-mcp-token
MCP_AUTH_MODE=oauth
MCP_RESOURCE_URL=http://localhost:8001/mcp
MCP_OAUTH_ISSUER=
MCP_OAUTH_AUDIENCE=
MCP_OAUTH_JWKS_URL=
MCP_OAUTH_REQUIRED_SCOPE=mcp:tools
API_PORT=8000 API_PORT=8000
MCP_PORT=8001 MCP_PORT=8001

1
.gitignore vendored
View File

@@ -4,6 +4,7 @@ __pycache__/
*.py[cod] *.py[cod]
*.db *.db
.pytest_cache/ .pytest_cache/
pytest-cache-files-*/
.coverage .coverage
htmlcov/ htmlcov/
postgres_data/ postgres_data/

View File

@@ -6,7 +6,7 @@
- `api`: FastAPI, REST API, HTML-админка, healthcheck. - `api`: FastAPI, REST API, HTML-админка, healthcheck.
- `worker`: weekly scheduler, который запускает парсинг по `CRAWL_CRON`. - `worker`: weekly scheduler, который запускает парсинг по `CRAWL_CRON`.
- `mcp`: HTTP MCP endpoint с bearer token. - `mcp`: HTTP MCP endpoint с OAuth/OIDC access token для внешних агентов или legacy static token для локального режима.
- `postgres`: основная БД. - `postgres`: основная БД.
Парсер использует фиксированный источник сотрудников, по умолчанию `https://miem.hse.ru/persons`. Для каждой карточки сохраняются ФИО, должности, год начала работы, контакты, идентификаторы, вкладки профиля, секции, публикации, курсы, ВКР, JSON-снапшот и сжатый HTML-снапшот. Ссылки обходятся только из меню профиля самого сотрудника (`person-menu`), например `#sci`, `#teaching`, `#main`. Парсер использует фиксированный источник сотрудников, по умолчанию `https://miem.hse.ru/persons`. Для каждой карточки сохраняются ФИО, должности, год начала работы, контакты, идентификаторы, вкладки профиля, секции, публикации, курсы, ВКР, JSON-снапшот и сжатый HTML-снапшот. Ссылки обходятся только из меню профиля самого сотрудника (`person-menu`), например `#sci`, `#teaching`, `#main`.
@@ -27,7 +27,13 @@ cp .env.example .env
- `CRAWL_LIMIT`: опциональный лимит профилей для тестового запуска. - `CRAWL_LIMIT`: опциональный лимит профилей для тестового запуска.
- `ADMIN_USERNAME`, `ADMIN_PASSWORD`: логин и пароль админки. - `ADMIN_USERNAME`, `ADMIN_PASSWORD`: логин и пароль админки.
- `SESSION_SECRET`: секрет подписи cookie. - `SESSION_SECRET`: секрет подписи cookie.
- `MCP_TOKEN`: bearer token для `/mcp`. - `MCP_TOKEN`: статический bearer token для legacy/local режима `MCP_AUTH_MODE=token`.
- `MCP_AUTH_MODE`: режим авторизации MCP: `oauth` для внешних агентов или `token` для локальной отладки.
- `MCP_RESOURCE_URL`: публичный URL MCP endpoint, например `https://example.com/mcp`.
- `MCP_OAUTH_ISSUER`: issuer внешнего OIDC-провайдера.
- `MCP_OAUTH_AUDIENCE`: ожидаемый `aud` в OAuth access token.
- `MCP_OAUTH_JWKS_URL`: JWKS endpoint; если не задан, используется `<issuer>/.well-known/jwks.json`.
- `MCP_OAUTH_REQUIRED_SCOPE`: scope для доступа к MCP tools, по умолчанию `mcp:tools`.
- `PARSER_USE_PLAYWRIGHT`: включение Playwright-рендера динамических вкладок. - `PARSER_USE_PLAYWRIGHT`: включение Playwright-рендера динамических вкладок.
## Локальный запуск ## Локальный запуск
@@ -82,7 +88,9 @@ curl -X POST http://localhost:8000/api/crawl-runs --cookie "miem_admin_session=.
## MCP ## MCP
Endpoint: `POST /mcp`, авторизация `Authorization: Bearer <MCP_TOKEN>`. Endpoint: `POST /mcp`, авторизация `Authorization: Bearer <token>`.
Для внешних ИИ-агентов используйте `MCP_AUTH_MODE=oauth`. В этом режиме статический `MCP_TOKEN` не принимается: клиент должен передать OAuth/OIDC access token с нужным scope.
Поддерживаемые tools: Поддерживаемые tools:
@@ -92,7 +100,7 @@ Endpoint: `POST /mcp`, авторизация `Authorization: Bearer <MCP_TOKEN>
- `list_employee_courses(profile_id_or_url)` - `list_employee_courses(profile_id_or_url)`
- `get_crawl_status()` - `get_crawl_status()`
Пример: Пример локального legacy-режима со статическим токеном:
```bash ```bash
curl http://localhost:8001/mcp \ curl http://localhost:8001/mcp \
@@ -101,6 +109,19 @@ curl http://localhost:8001/mcp \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}' -d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}'
``` ```
Для production OAuth/OIDC настройте внешний authorization server и включите режим `oauth`:
```env
MCP_AUTH_MODE=oauth
MCP_RESOURCE_URL=https://example.com/mcp
MCP_OAUTH_ISSUER=https://auth.example.com
MCP_OAUTH_AUDIENCE=miem-mcp
MCP_OAUTH_JWKS_URL=https://auth.example.com/.well-known/jwks.json
MCP_OAUTH_REQUIRED_SCOPE=mcp:tools
```
MCP server работает как OAuth protected resource: он не выдает токены, а проверяет JWT access token по JWKS, `issuer`, `audience`, сроку действия и scope. Metadata для MCP-клиентов доступна по `GET /.well-known/oauth-protected-resource`.
## Обслуживание ## Обслуживание
```bash ```bash
@@ -110,4 +131,4 @@ docker compose exec postgres pg_dump -U miem miem_workers > backup.sql
docker compose down docker compose down
``` ```
Версия сервиса: `0.2.4`. Админка всегда показывает версии backend и frontend в footer. Версия сервиса: `0.4.0`. Админка всегда показывает версии backend и frontend в footer.

View File

@@ -1,14 +1,21 @@
from fastapi import APIRouter, BackgroundTasks, Depends, Form, Request from fastapi import APIRouter, BackgroundTasks, Depends, Form, Request
from fastapi.responses import HTMLResponse, RedirectResponse from fastapi.responses import HTMLResponse, RedirectResponse
from fastapi.templating import Jinja2Templates from fastapi.templating import Jinja2Templates
from sqlalchemy import desc, func, or_, select from sqlalchemy import desc, func, select
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
from app.config import Settings, get_settings from app.config import Settings, get_settings
from app.db import SessionLocal, get_db from app.db import SessionLocal, get_db
from app.models import CrawlError, CrawlRun, Employee from app.models import CrawlError, CrawlRun, Employee
from app.security import SESSION_COOKIE, require_admin, sign_session, verify_admin from app.security import SESSION_COOKIE, require_admin, sign_session, verify_admin
from app.services.admin_data import employee_detail_payload, list_employees_page, run_payload, stats_payload from app.services.admin_data import (
employee_detail_payload,
format_admin_datetime,
list_employees_page,
run_detail_payload,
run_payload,
stats_payload,
)
from app.services.crawl_control import get_running_run, run_crawl_if_idle from app.services.crawl_control import get_running_run, run_crawl_if_idle
from app.version import BACKEND_VERSION, FRONTEND_VERSION from app.version import BACKEND_VERSION, FRONTEND_VERSION
@@ -22,8 +29,9 @@ def dashboard(request: Request, db: Session = Depends(get_db), settings: Setting
counts = stats_payload(db) counts = stats_payload(db)
counts["runs"] = db.scalar(select(func.count()).select_from(CrawlRun)) or 0 counts["runs"] = db.scalar(select(func.count()).select_from(CrawlRun)) or 0
counts["errors"] = db.scalar(select(func.count()).select_from(CrawlError)) or 0 counts["errors"] = db.scalar(select(func.count()).select_from(CrawlError)) or 0
runs = db.scalars(select(CrawlRun).order_by(desc(CrawlRun.started_at)).limit(10)).all() run_models = db.scalars(select(CrawlRun).order_by(desc(CrawlRun.started_at)).limit(10)).all()
return _render(request, "dashboard.html", {"counts": counts, "runs": runs, "latest_run": run_payload(runs[0]) if runs else None}) runs = [run_payload(run) for run in run_models]
return _render(request, "dashboard.html", {"counts": counts, "runs": runs, "latest_run": runs[0] if runs else None})
@router.get("/login", response_class=HTMLResponse) @router.get("/login", response_class=HTMLResponse)
@@ -57,18 +65,10 @@ def employees(
request: Request, request: Request,
status: str | None = None, status: str | None = None,
q: str | None = None, q: str | None = None,
db: Session = Depends(get_db),
settings: Settings = Depends(get_settings), settings: Settings = Depends(get_settings),
): ):
require_admin(request, settings) require_admin(request, settings)
stmt = select(Employee) return RedirectResponse("/admin/directory", status_code=303)
if status:
stmt = stmt.where(Employee.status == status)
if q:
pattern = f"%{q}%"
stmt = stmt.where(or_(Employee.full_name.ilike(pattern), Employee.canonical_url.ilike(pattern)))
items = db.scalars(stmt.order_by(Employee.full_name).limit(200)).all()
return _render(request, "employees.html", {"employees": items, "status": status or "", "q": q or ""})
@router.get("/directory", response_class=HTMLResponse) @router.get("/directory", response_class=HTMLResponse)
@@ -115,7 +115,7 @@ def directory(
"has_email": has_email or "", "has_email": has_email or "",
"sort": sort, "sort": sort,
"direction": direction, "direction": direction,
"limit": limit, "limit": page["limit"],
"offset": offset, "offset": offset,
}, },
}, },
@@ -133,7 +133,14 @@ def employee_detail(
employee = db.get(Employee, employee_id) employee = db.get(Employee, employee_id)
if not employee: if not employee:
return RedirectResponse("/admin/employees", status_code=303) return RedirectResponse("/admin/employees", status_code=303)
snapshots = sorted(employee.snapshots, key=lambda item: item.captured_at, reverse=True)[:20] snapshots = [
{
"captured_display": format_admin_datetime(snapshot.captured_at),
"checksum": snapshot.checksum,
"parser_version": snapshot.parser_version,
}
for snapshot in sorted(employee.snapshots, key=lambda item: item.captured_at, reverse=True)[:20]
]
return _render( return _render(
request, request,
"employee_detail.html", "employee_detail.html",
@@ -144,11 +151,26 @@ def employee_detail(
@router.get("/runs", response_class=HTMLResponse) @router.get("/runs", response_class=HTMLResponse)
def runs(request: Request, db: Session = Depends(get_db), settings: Settings = Depends(get_settings)): def runs(request: Request, db: Session = Depends(get_db), settings: Settings = Depends(get_settings)):
require_admin(request, settings) require_admin(request, settings)
items = db.scalars(select(CrawlRun).order_by(desc(CrawlRun.started_at)).limit(50)).all() run_models = db.scalars(select(CrawlRun).order_by(desc(CrawlRun.started_at)).limit(50)).all()
items = [run_payload(run) for run in run_models]
errors = db.scalars(select(CrawlError).order_by(desc(CrawlError.created_at)).limit(50)).all() errors = db.scalars(select(CrawlError).order_by(desc(CrawlError.created_at)).limit(50)).all()
return _render(request, "runs.html", {"runs": items, "errors": errors}) return _render(request, "runs.html", {"runs": items, "errors": errors})
@router.get("/runs/{run_id}", response_class=HTMLResponse)
def run_detail(
run_id: int,
request: Request,
db: Session = Depends(get_db),
settings: Settings = Depends(get_settings),
):
require_admin(request, settings)
run = db.get(CrawlRun, run_id)
if not run:
return RedirectResponse("/admin/runs", status_code=303)
return _render(request, "run_detail.html", {"run": run_detail_payload(db, run)})
@router.post("/runs") @router.post("/runs")
def trigger_run( def trigger_run(
request: Request, request: Request,

View File

@@ -8,7 +8,7 @@ from app.config import Settings, get_settings
from app.db import SessionLocal, get_db from app.db import SessionLocal, get_db
from app.models import CrawlRun, Employee from app.models import CrawlRun, Employee
from app.security import require_admin from app.security import require_admin
from app.services.admin_data import employee_display_payload, list_employees_page, run_payload, stats_payload from app.services.admin_data import employee_display_payload, list_employees_page, run_detail_payload, run_payload, stats_payload
from app.services.crawl_control import get_running_run, run_crawl_if_idle from app.services.crawl_control import get_running_run, run_crawl_if_idle
from app.version import BACKEND_VERSION, FRONTEND_VERSION from app.version import BACKEND_VERSION, FRONTEND_VERSION
@@ -88,6 +88,20 @@ def latest_crawl_run(
return {"running": run_payload(running), "latest": run_payload(latest)} return {"running": run_payload(running), "latest": run_payload(latest)}
@router.get("/crawl-runs/{run_id}")
def get_crawl_run(
run_id: int,
request: Request,
db: Session = Depends(get_db),
settings: Settings = Depends(get_settings),
) -> dict:
require_admin(request, settings)
run = db.get(CrawlRun, run_id)
if not run:
return {"error": "not_found"}
return run_detail_payload(db, run) or {"error": "not_found"}
@router.post("/crawl-runs") @router.post("/crawl-runs")
def trigger_crawl( def trigger_crawl(
request: Request, request: Request,

View File

@@ -1,5 +1,7 @@
from functools import lru_cache from functools import lru_cache
from pydantic import Field from typing import Literal
from pydantic import Field, field_validator
from pydantic_settings import BaseSettings, SettingsConfigDict from pydantic_settings import BaseSettings, SettingsConfigDict
@@ -18,6 +20,27 @@ class Settings(BaseSettings):
admin_password: str = "admin" admin_password: str = "admin"
session_secret: str = Field(default="dev-session-secret", min_length=8) session_secret: str = Field(default="dev-session-secret", min_length=8)
mcp_token: str = "dev-mcp-token" mcp_token: str = "dev-mcp-token"
mcp_auth_mode: Literal["token", "oauth"] = "oauth"
mcp_resource_url: str = "http://localhost:8001/mcp"
mcp_oauth_issuer: str = ""
mcp_oauth_audience: str = ""
mcp_oauth_jwks_url: str = ""
mcp_oauth_required_scope: str = "mcp:tools"
@field_validator("crawl_limit", mode="before")
@classmethod
def empty_crawl_limit_as_none(cls, value):
if value == "":
return None
return value
def oauth_jwks_url(self) -> str:
if self.mcp_oauth_jwks_url:
return self.mcp_oauth_jwks_url
issuer = self.mcp_oauth_issuer.rstrip("/")
if not issuer:
return ""
return f"{issuer}/.well-known/jwks.json"
@lru_cache @lru_cache

View File

@@ -4,6 +4,7 @@ from fastapi.staticfiles import StaticFiles
from app.admin import router as admin_router from app.admin import router as admin_router
from app.api import router as api_router from app.api import router as api_router
from app.db import init_db from app.db import init_db
from app.mcp import metadata_router as mcp_metadata_router
from app.mcp import router as mcp_router from app.mcp import router as mcp_router
from app.version import BACKEND_VERSION from app.version import BACKEND_VERSION
@@ -12,6 +13,7 @@ app.mount("/static", StaticFiles(directory="app/static"), name="static")
app.include_router(api_router) app.include_router(api_router)
app.include_router(admin_router) app.include_router(admin_router)
app.include_router(mcp_router) app.include_router(mcp_router)
app.include_router(mcp_metadata_router)
@app.on_event("startup") @app.on_event("startup")

View File

@@ -7,9 +7,12 @@ from sqlalchemy.orm import Session
from app.config import Settings, get_settings from app.config import Settings, get_settings
from app.db import get_db from app.db import get_db
from app.models import CrawlRun, Employee from app.models import CrawlRun, Employee
from app.security import require_mcp_token from app.security import mcp_protected_resource_metadata, require_mcp_auth
from app.services.admin_data import run_detail_payload
from app.version import BACKEND_VERSION
router = APIRouter(prefix="/mcp") router = APIRouter(prefix="/mcp")
metadata_router = APIRouter()
TOOLS = [ TOOLS = [
@@ -46,6 +49,15 @@ TOOLS = [
"description": "Return the latest crawl run status.", "description": "Return the latest crawl run status.",
"inputSchema": {"type": "object", "properties": {}}, "inputSchema": {"type": "object", "properties": {}},
}, },
{
"name": "get_crawl_run_details",
"description": "Return detailed employee changes and errors for one crawl run.",
"inputSchema": {
"type": "object",
"properties": {"run_id": {"type": "integer"}},
"required": ["run_id"],
},
},
] ]
@@ -55,7 +67,7 @@ async def mcp_http(
db: Session = Depends(get_db), db: Session = Depends(get_db),
settings: Settings = Depends(get_settings), settings: Settings = Depends(get_settings),
) -> dict: ) -> dict:
require_mcp_token(request, settings) require_mcp_auth(request, settings)
payload = await request.json() payload = await request.json()
method = payload.get("method") method = payload.get("method")
request_id = payload.get("id") request_id = payload.get("id")
@@ -65,7 +77,7 @@ async def mcp_http(
if method == "initialize": if method == "initialize":
result = { result = {
"protocolVersion": "2024-11-05", "protocolVersion": "2024-11-05",
"serverInfo": {"name": "miem-employees", "version": "0.1.0"}, "serverInfo": {"name": "miem-employees", "version": BACKEND_VERSION},
"capabilities": {"tools": {}}, "capabilities": {"tools": {}},
} }
elif method == "tools/list": elif method == "tools/list":
@@ -94,6 +106,9 @@ def _call_tool(db: Session, name: str, arguments: dict) -> dict:
if name == "get_crawl_status": if name == "get_crawl_status":
run = db.scalar(select(CrawlRun).order_by(desc(CrawlRun.started_at)).limit(1)) run = db.scalar(select(CrawlRun).order_by(desc(CrawlRun.started_at)).limit(1))
return _tool_response(_run_payload(run) if run else {"status": "never_run"}) return _tool_response(_run_payload(run) if run else {"status": "never_run"})
if name == "get_crawl_run_details":
run = db.get(CrawlRun, int(arguments["run_id"]))
return _tool_response(run_detail_payload(db, run) if run else {"error": "not_found"})
raise ValueError(f"Unknown tool: {name}") raise ValueError(f"Unknown tool: {name}")
@@ -168,3 +183,8 @@ def _run_payload(run: CrawlRun) -> dict:
def _tool_response(data: object) -> dict: def _tool_response(data: object) -> dict:
return {"content": [{"type": "text", "text": json.dumps(data, ensure_ascii=False, default=str)}]} return {"content": [{"type": "text", "text": json.dumps(data, ensure_ascii=False, default=str)}]}
@metadata_router.get("/.well-known/oauth-protected-resource")
def oauth_protected_resource(settings: Settings = Depends(get_settings)) -> dict:
return mcp_protected_resource_metadata(settings)

View File

@@ -41,6 +41,7 @@ class Employee(Base):
snapshots: Mapped[list["EmployeeSnapshot"]] = relationship(back_populates="employee") snapshots: Mapped[list["EmployeeSnapshot"]] = relationship(back_populates="employee")
tabs: Mapped[list["ProfileTab"]] = relationship(back_populates="employee", cascade="all, delete-orphan") tabs: Mapped[list["ProfileTab"]] = relationship(back_populates="employee", cascade="all, delete-orphan")
crawl_run_changes: Mapped[list["CrawlRunEmployeeChange"]] = relationship(back_populates="employee")
class EmployeeSnapshot(Base): class EmployeeSnapshot(Base):
@@ -74,6 +75,31 @@ class CrawlRun(Base):
dismissed_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False) dismissed_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
message: Mapped[str | None] = mapped_column(Text) message: Mapped[str | None] = mapped_column(Text)
employee_changes: Mapped[list["CrawlRunEmployeeChange"]] = relationship(back_populates="crawl_run")
class CrawlRunEmployeeChange(Base):
__tablename__ = "crawl_run_employee_changes"
__table_args__ = (
Index("ix_crawl_run_employee_changes_run_id", "crawl_run_id"),
Index("ix_crawl_run_employee_changes_employee_id", "employee_id"),
Index("ix_crawl_run_employee_changes_change_type", "change_type"),
)
id: Mapped[int] = mapped_column(Integer, primary_key=True)
crawl_run_id: Mapped[int] = mapped_column(ForeignKey("crawl_runs.id"), nullable=False)
employee_id: Mapped[int | None] = mapped_column(ForeignKey("employees.id"))
profile_key: Mapped[str] = mapped_column(String(255), nullable=False)
profile_url: Mapped[str] = mapped_column(Text, nullable=False)
full_name: Mapped[str | None] = mapped_column(Text)
change_type: Mapped[str] = mapped_column(String(32), nullable=False)
profile_available: Mapped[bool | None] = mapped_column()
message: Mapped[str | None] = mapped_column(Text)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow, nullable=False)
crawl_run: Mapped[CrawlRun] = relationship(back_populates="employee_changes")
employee: Mapped[Employee | None] = relationship(back_populates="crawl_run_changes")
class CrawlError(Base): class CrawlError(Base):
__tablename__ = "crawl_errors" __tablename__ = "crawl_errors"

View File

@@ -164,6 +164,7 @@ def parse_person_profile(
header = extract_person_header(soup, normalized_url) header = extract_person_header(soup, normalized_url)
tabs = extract_person_tabs(soup, normalized_url) tabs = extract_person_tabs(soup, normalized_url)
sections = extract_sections(soup, normalized_url) sections = extract_sections(soup, normalized_url)
sections = enrich_sections_from_hse_widgets(session, soup, normalized_url, headers, timeout, sections)
internal_links = [tab["href"] for tab in tabs if tab.get("href")] internal_links = [tab["href"] for tab in tabs if tab.get("href")]
return { return {
@@ -183,6 +184,25 @@ def parse_person_profile(
} }
def enrich_sections_from_hse_widgets(
session: Session,
soup: BeautifulSoup,
source_url: str,
headers: dict[str, str],
timeout: int,
sections: list[dict],
) -> list[dict]:
enriched = list(sections)
publications = _load_widget_publications(session, soup, headers, timeout)
if publications:
enriched = _upsert_publications_section(enriched, publications)
theses = _load_widget_graduation_theses(session, soup, source_url, headers, timeout)
if theses:
enriched = _upsert_graduation_theses_section(enriched, theses)
return enriched
def _render_with_playwright(source_url: str, fallback_html: str) -> str: def _render_with_playwright(source_url: str, fallback_html: str) -> str:
try: try:
from playwright.sync_api import sync_playwright from playwright.sync_api import sync_playwright
@@ -206,6 +226,89 @@ def _render_with_playwright(source_url: str, fallback_html: str) -> str:
return fallback_html return fallback_html
def _load_widget_publications(session: Session, soup: BeautifulSoup, headers: dict[str, str], timeout: int) -> list[dict]:
script = soup.select_one('script[data-widget-name="AuthorSearch"][data-author]')
if not script:
return []
author_id = normalize_ws(script.get("data-author"))
if not author_id:
return []
publications = []
page_id = 1
per_page = 100
while page_id <= 20:
payload = {
"type": "ANY",
"filterParams": (
f'"acceptLanguage":"ru"|"fullTextPublicEnabled": 1|'
f'"pubsAuthor": {author_id}|"widgetName": "AuthorSearch"'
),
"paginationParams": {
"publsSort": ["TITLE_ASC"],
"publsCount": per_page,
"pageId": page_id,
},
}
try:
response = session.post(
"https://publications.hse.ru/api/searchPubs",
json=payload,
headers=headers,
timeout=timeout,
)
response.raise_for_status()
data = response.json()
except Exception:
return publications
result = data.get("result") if isinstance(data, dict) else {}
items = result.get("items") if isinstance(result, dict) else []
if not isinstance(items, list) or not items:
break
publications.extend(_normalize_publication_item(item) for item in items if isinstance(item, dict))
total = int(result.get("total") or 0)
if not result.get("more") and len(publications) >= total:
break
page_id += 1
return _dedupe_publications(publications)
def _load_widget_graduation_theses(
session: Session,
soup: BeautifulSoup,
source_url: str,
headers: dict[str, str],
timeout: int,
) -> list[dict]:
script = soup.select_one('script[src*="/n/stat/vkr/app.js"][data-person-id]')
if not script:
return []
person_id = normalize_ws(script.get("data-person-id"))
api_url = normalize_ws(script.get("data-api-url")) or "/n/vkr/api/"
if not person_id:
return []
request_headers = {**headers, "x-portal-language": "ru"}
try:
response = session.get(
urljoin(source_url, api_url),
params={"supervisorId": person_id},
headers=request_headers,
timeout=timeout,
)
response.raise_for_status()
data = response.json()
except Exception:
return []
items = data.get("data") if isinstance(data, dict) else []
if not isinstance(items, list):
return []
return [_normalize_vkr_item(item, source_url) for item in items if isinstance(item, dict)]
def _collect_between_h2(start_h2: Tag) -> list[Tag | NavigableString | str]: def _collect_between_h2(start_h2: Tag) -> list[Tag | NavigableString | str]:
nodes = [] nodes = []
for sibling in start_h2.next_siblings: for sibling in start_h2.next_siblings:
@@ -353,6 +456,122 @@ def _parse_vkr_items(nodes: list) -> list[str]:
return [item for item in dict.fromkeys(items) if item] return [item for item in dict.fromkeys(items) if item]
def _normalize_publication_item(item: dict) -> dict:
publication_id = str(item.get("id") or "").strip()
title = _html_to_text(item.get("title"))
year = item.get("year")
publication_type = str(item.get("type") or "").strip() or None
description = item.get("description") if isinstance(item.get("description"), dict) else {}
short_description = _localized_value(description.get("short")) or _localized_value(description.get("shortLeft"))
text = normalize_ws(" ".join(part for part in [title, str(year or ""), short_description] if part))
return {
"id": publication_id or None,
"title": title or publication_id,
"year": year,
"type": publication_type,
"url": f"https://publications.hse.ru/view/{publication_id}" if publication_id else None,
"text": text or title or publication_id,
}
def _normalize_vkr_item(item: dict, source_url: str) -> dict:
thesis_id = item.get("id")
program = item.get("learnProgram") if isinstance(item.get("learnProgram"), dict) else {}
org_unit = item.get("orgUnit") if isinstance(item.get("orgUnit"), dict) else {}
supervisors = []
for supervisor in item.get("supervisors") or []:
if not isinstance(supervisor, dict):
continue
name = normalize_ws(supervisor.get("name"))
url = normalize_ws(supervisor.get("url"))
if name or url:
supervisors.append({"name": name or url, "url": url or None})
return {
"id": thesis_id,
"student": normalize_ws(item.get("student")),
"title": normalize_ws(item.get("title")),
"defense_year": item.get("year"),
"level": normalize_ws(item.get("level")),
"rating": item.get("rating"),
"project_url": urljoin(source_url, f"/edu/vkr/{thesis_id}") if thesis_id else None,
"program": normalize_ws(program.get("title")),
"program_url": urljoin(source_url, program.get("url")) if program.get("url") else None,
"org_unit": normalize_ws(org_unit.get("title")),
"org_unit_url": urljoin(source_url, org_unit.get("url")) if org_unit.get("url") else None,
"supervisors": supervisors,
"text": normalize_ws(" ".join(str(part) for part in [item.get("student"), item.get("title"), item.get("year")] if part)),
}
def _upsert_publications_section(sections: list[dict], publications: list[dict]) -> list[dict]:
merged = []
inserted = False
for section in sections:
if section.get("type") != "publications":
merged.append(section)
continue
existing = section.get("publications") or []
section = {
**section,
"publications_count": max(section.get("publications_count") or 0, len(publications)),
"publications": _dedupe_publications([*existing, *publications]),
}
section["items"] = [item["text"] for item in section["publications"] if item.get("text")]
merged.append(section)
inserted = True
if not inserted:
merged.append(
{
"title": "Публикации и исследования",
"slug": "publikacii_i_issledovaniya",
"type": "publications",
"raw_text": "",
"paragraphs": [],
"items": [item["text"] for item in publications if item.get("text")],
"links": [],
"publications_count": len(publications),
"publications": publications,
}
)
return merged
def _upsert_graduation_theses_section(sections: list[dict], theses: list[dict]) -> list[dict]:
section = {
"title": "Выпускные квалификационные работы студентов НИУ ВШЭ",
"slug": "vypusknye_kvalifikacionnye_raboty_studentov_niu_vshe",
"type": "graduation_theses",
"raw_text": "",
"paragraphs": [],
"items": [item["text"] for item in theses if item.get("text")],
"links": [{"text": item["title"], "url": item["project_url"]} for item in theses if item.get("title") and item.get("project_url")],
"theses_count": len(theses),
"theses": theses,
}
return [item for item in sections if item.get("type") != "graduation_theses"] + [section]
def _dedupe_publications(items: list[dict]) -> list[dict]:
seen = set()
unique = []
for item in items:
key = item.get("id") or item.get("url") or item.get("title")
if key and key not in seen:
seen.add(key)
unique.append(item)
return unique
def _html_to_text(value: object) -> str:
return normalize_ws(BeautifulSoup(str(value or ""), "html.parser").get_text(" ", strip=True))
def _localized_value(value: object) -> str:
if isinstance(value, dict):
return normalize_ws(value.get("ru") or value.get("publ") or value.get("en"))
return normalize_ws(str(value or ""))
def _slugify(value: str) -> str: def _slugify(value: str) -> str:
cleaned = re.sub(r"[^\w\s-]", "", value.lower(), flags=re.UNICODE) cleaned = re.sub(r"[^\w\s-]", "", value.lower(), flags=re.UNICODE)
return re.sub(r"[-\s]+", "_", cleaned).strip("_") or "section" return re.sub(r"[-\s]+", "_", cleaned).strip("_") or "section"

View File

@@ -3,7 +3,10 @@ import hashlib
import hmac import hmac
import json import json
import time import time
from functools import lru_cache
import jwt
from jwt import PyJWKClient, PyJWTError
from fastapi import HTTPException, Request, status from fastapi import HTTPException, Request, status
from app.config import Settings from app.config import Settings
@@ -46,7 +49,91 @@ def require_admin(request: Request, settings: Settings) -> str:
return username return username
def require_mcp_token(request: Request, settings: Settings) -> None: def require_mcp_auth(request: Request, settings: Settings) -> None:
auth = request.headers.get("authorization", "") auth = request.headers.get("authorization", "")
if not auth.startswith("Bearer ") or not hmac.compare_digest(auth.removeprefix("Bearer ").strip(), settings.mcp_token): if not auth.startswith("Bearer "):
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Invalid MCP token") raise _mcp_unauthorized(settings, "Missing bearer token")
token = auth.removeprefix("Bearer ").strip()
if _mcp_static_token_allowed(settings) and hmac.compare_digest(token, settings.mcp_token):
return
if _mcp_oauth_allowed(settings):
_validate_mcp_oauth_token(token, settings)
return
raise _mcp_unauthorized(settings, "Invalid MCP token")
def require_mcp_token(request: Request, settings: Settings) -> None:
require_mcp_auth(request, settings)
def mcp_protected_resource_metadata(settings: Settings) -> dict:
authorization_servers = [settings.mcp_oauth_issuer.rstrip("/")] if settings.mcp_oauth_issuer else []
return {
"resource": settings.mcp_resource_url,
"authorization_servers": authorization_servers,
"bearer_methods_supported": ["header"],
"scopes_supported": [settings.mcp_oauth_required_scope],
"resource_documentation": settings.mcp_resource_url,
}
def _mcp_static_token_allowed(settings: Settings) -> bool:
return settings.mcp_auth_mode == "token"
def _mcp_oauth_allowed(settings: Settings) -> bool:
return settings.mcp_auth_mode == "oauth"
def _validate_mcp_oauth_token(token: str, settings: Settings) -> None:
if not settings.mcp_oauth_issuer or not settings.mcp_oauth_audience or not settings.oauth_jwks_url():
raise _mcp_unauthorized(settings, "MCP OAuth is not configured")
try:
signing_key = _get_mcp_oauth_signing_key(token, settings).key
claims = jwt.decode(
token,
signing_key,
algorithms=["RS256", "RS384", "RS512", "ES256", "ES384", "ES512"],
audience=settings.mcp_oauth_audience,
issuer=settings.mcp_oauth_issuer.rstrip("/"),
)
except PyJWTError as exc:
raise _mcp_unauthorized(settings, "Invalid OAuth access token") from exc
if not _claims_have_scope(claims, settings.mcp_oauth_required_scope):
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Missing required MCP OAuth scope")
def _claims_have_scope(claims: dict, required_scope: str) -> bool:
scopes: set[str] = set()
scope = claims.get("scope")
if isinstance(scope, str):
scopes.update(scope.split())
scp = claims.get("scp")
if isinstance(scp, str):
scopes.update(scp.split())
elif isinstance(scp, list):
scopes.update(str(item) for item in scp)
return required_scope in scopes
@lru_cache(maxsize=16)
def _get_jwk_client(jwks_url: str) -> PyJWKClient:
return PyJWKClient(jwks_url)
def _get_mcp_oauth_signing_key(token: str, settings: Settings):
return _get_jwk_client(settings.oauth_jwks_url()).get_signing_key_from_jwt(token)
def _mcp_unauthorized(settings: Settings, detail: str) -> HTTPException:
headers = {}
if _mcp_oauth_allowed(settings):
headers["WWW-Authenticate"] = f'Bearer resource_metadata="{_mcp_metadata_url(settings)}"'
return HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail=detail, headers=headers)
def _mcp_metadata_url(settings: Settings) -> str:
resource_url = settings.mcp_resource_url.rstrip("/")
base_url = resource_url[: -len("/mcp")] if resource_url.endswith("/mcp") else resource_url
return f"{base_url}/.well-known/oauth-protected-resource"

View File

@@ -3,11 +3,12 @@ from __future__ import annotations
from datetime import date, datetime, time from datetime import date, datetime, time
from math import ceil from math import ceil
from typing import Any from typing import Any
from zoneinfo import ZoneInfo
from sqlalchemy import Select, Text, and_, desc, func, or_, select from sqlalchemy import Select, Text, and_, desc, func, or_, select
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
from app.models import CrawlRun, Employee from app.models import CrawlError, CrawlRun, CrawlRunEmployeeChange, Employee
EMPLOYEE_SORTS = { EMPLOYEE_SORTS = {
"full_name": Employee.full_name, "full_name": Employee.full_name,
@@ -30,6 +31,7 @@ def employee_display_payload(employee: Employee) -> dict[str, Any]:
"id": employee.id, "id": employee.id,
"full_name": employee.full_name, "full_name": employee.full_name,
"status": employee.status, "status": employee.status,
"status_display": _employee_status_display(employee.status),
"canonical_url": employee.canonical_url, "canonical_url": employee.canonical_url,
"positions": positions, "positions": positions,
"positions_text": "; ".join(positions), "positions_text": "; ".join(positions),
@@ -44,6 +46,9 @@ def employee_display_payload(employee: Employee) -> dict[str, Any]:
"first_seen_at": employee.first_seen_at.isoformat() if employee.first_seen_at else None, "first_seen_at": employee.first_seen_at.isoformat() if employee.first_seen_at else None,
"last_seen_at": employee.last_seen_at.isoformat() if employee.last_seen_at else None, "last_seen_at": employee.last_seen_at.isoformat() if employee.last_seen_at else None,
"dismissed_at": employee.dismissed_at.isoformat() if employee.dismissed_at else None, "dismissed_at": employee.dismissed_at.isoformat() if employee.dismissed_at else None,
"first_seen_display": format_admin_datetime(employee.first_seen_at),
"last_seen_display": format_admin_datetime(employee.last_seen_at),
"dismissed_display": format_admin_datetime(employee.dismissed_at),
} }
@@ -107,7 +112,7 @@ def list_employees_page(
limit: int = 50, limit: int = 50,
offset: int = 0, offset: int = 0,
) -> dict[str, Any]: ) -> dict[str, Any]:
limit = max(1, min(limit, 200)) limit = limit if limit in {25, 50, 100} else 50
offset = max(0, offset) offset = max(0, offset)
base_stmt = build_employee_query( base_stmt = build_employee_query(
status=status, status=status,
@@ -121,7 +126,7 @@ def list_employees_page(
order = desc(sort_column) if direction == "desc" else sort_column order = desc(sort_column) if direction == "desc" else sort_column
employees = db.scalars(base_stmt.order_by(order).limit(limit).offset(offset)).all() employees = db.scalars(base_stmt.order_by(order).limit(limit).offset(offset)).all()
return { return {
"items": [employee_display_payload(employee) for employee in employees], "employees": [employee_display_payload(employee) for employee in employees],
"total": total, "total": total,
"limit": limit, "limit": limit,
"offset": offset, "offset": offset,
@@ -154,8 +159,11 @@ def run_payload(run: CrawlRun | None) -> dict[str, Any] | None:
"id": run.id, "id": run.id,
"source_url": run.source_url, "source_url": run.source_url,
"status": run.status, "status": run.status,
"status_display": _run_status_display(run.status),
"started_at": run.started_at.isoformat() if run.started_at else None, "started_at": run.started_at.isoformat() if run.started_at else None,
"finished_at": run.finished_at.isoformat() if run.finished_at else None, "finished_at": run.finished_at.isoformat() if run.finished_at else None,
"started_display": format_admin_datetime(run.started_at),
"finished_display": format_admin_datetime(run.finished_at),
"found_count": run.found_count, "found_count": run.found_count,
"parsed_count": run.parsed_count, "parsed_count": run.parsed_count,
"new_count": run.new_count, "new_count": run.new_count,
@@ -167,6 +175,97 @@ def run_payload(run: CrawlRun | None) -> dict[str, Any] | None:
} }
def run_detail_payload(db: Session, run: CrawlRun | None) -> dict[str, Any] | None:
if not run:
return None
changes = db.scalars(
select(CrawlRunEmployeeChange)
.where(CrawlRunEmployeeChange.crawl_run_id == run.id)
.order_by(CrawlRunEmployeeChange.created_at, CrawlRunEmployeeChange.id)
).all()
errors = db.scalars(select(CrawlError).where(CrawlError.crawl_run_id == run.id).order_by(CrawlError.created_at)).all()
grouped_changes = {"new": [], "missing_from_source": [], "dismissed": []}
for change in changes:
grouped_changes.setdefault(change.change_type, []).append(_change_payload(change))
return {
**(run_payload(run) or {}),
"changes_detail_available": bool(changes),
"changes": grouped_changes,
"errors": [_crawl_error_payload(error) for error in errors],
}
def format_admin_datetime(value: Any) -> str:
if not value:
return "Не указано"
if isinstance(value, str):
try:
value = datetime.fromisoformat(value.replace("Z", "+00:00"))
except ValueError:
return value
if not isinstance(value, datetime):
return str(value)
if value.tzinfo:
value = value.astimezone(ZoneInfo("Europe/Moscow"))
return value.strftime("%d.%m.%Y %H:%M")
def _employee_status_display(status: str | None) -> str:
labels = {"active": "Работает", "dismissed": "Уволен"}
return labels.get(status or "", status or "Не указано")
def _run_status_display(status: str | None) -> str:
labels = {"running": "Выполняется", "completed": "Завершен", "failed": "Ошибка"}
return labels.get(status or "", status or "Не указано")
def _change_payload(change: CrawlRunEmployeeChange) -> dict[str, Any]:
return {
"id": change.id,
"employee_id": change.employee_id,
"profile_key": change.profile_key,
"profile_url": change.profile_url,
"full_name": change.full_name,
"change_type": change.change_type,
"change_type_display": _change_type_display(change.change_type),
"profile_available": change.profile_available,
"profile_available_display": _profile_available_display(change.profile_available),
"message": change.message,
"created_at": change.created_at.isoformat() if change.created_at else None,
"created_display": format_admin_datetime(change.created_at),
}
def _crawl_error_payload(error: CrawlError) -> dict[str, Any]:
return {
"id": error.id,
"crawl_run_id": error.crawl_run_id,
"profile_url": error.profile_url,
"error_type": error.error_type,
"message": error.message,
"created_at": error.created_at.isoformat() if error.created_at else None,
"created_display": format_admin_datetime(error.created_at),
}
def _change_type_display(change_type: str | None) -> str:
labels = {
"new": "Новый",
"missing_from_source": "Потеряшка",
"dismissed": "Уволен",
}
return labels.get(change_type or "", change_type or "Не указано")
def _profile_available_display(value: bool | None) -> str:
if value is True:
return "Профиль доступен"
if value is False:
return "Профиль недоступен"
return "Не проверялось"
def _count_section_items(sections: list[dict[str, Any]], section_type: str) -> int: def _count_section_items(sections: list[dict[str, Any]], section_type: str) -> int:
total = 0 total = 0
for section in sections: for section in sections:
@@ -243,11 +342,13 @@ def _normalize_section(section: Any) -> dict[str, Any]:
"type": section_type, "type": section_type,
"raw_text": raw_text, "raw_text": raw_text,
"paragraphs": paragraphs, "paragraphs": paragraphs,
"items": items, "list_items": items,
"links": _normalize_links(section.get("links")), "links": _normalize_links(section.get("links")),
"year_entries": _normalize_year_entries(section.get("year_entries")), "year_entries": _normalize_year_entries(section.get("year_entries")),
"publications": _normalize_publications(section.get("publications")), "publications": _normalize_publications(section.get("publications")),
"publications_count": section.get("publications_count"), "publications_count": section.get("publications_count"),
"theses": _normalize_theses(section.get("theses")),
"theses_count": section.get("theses_count"),
"academic_year": section.get("academic_year"), "academic_year": section.get("academic_year"),
"courses": _normalize_courses(section.get("courses")), "courses": _normalize_courses(section.get("courses")),
"table": _normalize_table(section.get("table")), "table": _normalize_table(section.get("table")),
@@ -316,6 +417,35 @@ def _normalize_courses(items: Any) -> list[dict[str, str | None]]:
return normalized return normalized
def _normalize_theses(items: Any) -> list[dict[str, Any]]:
normalized = []
if not isinstance(items, list):
return normalized
for item in items:
if not isinstance(item, dict):
continue
title = str(item.get("title") or "").strip()
student = str(item.get("student") or "").strip()
if not title and not student:
continue
normalized.append(
{
"id": item.get("id"),
"student": student,
"title": title,
"defense_year": item.get("defense_year") or item.get("year"),
"level": str(item.get("level") or "").strip(),
"rating": item.get("rating"),
"project_url": str(item.get("project_url") or "").strip() or None,
"program": str(item.get("program") or "").strip(),
"program_url": str(item.get("program_url") or "").strip() or None,
"org_unit": str(item.get("org_unit") or "").strip(),
"org_unit_url": str(item.get("org_unit_url") or "").strip() or None,
}
)
return normalized
def _normalize_table(table: Any) -> dict[str, Any] | None: def _normalize_table(table: Any) -> dict[str, Any] | None:
if not isinstance(table, dict): if not isinstance(table, dict):
return None return None

View File

@@ -9,7 +9,7 @@ from sqlalchemy import select
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
from app.config import Settings from app.config import Settings
from app.models import CrawlError, CrawlRun, Employee, EmployeeSnapshot, ParserSource, ProfileTab from app.models import CrawlError, CrawlRun, CrawlRunEmployeeChange, Employee, EmployeeSnapshot, ParserSource, ProfileTab
from app.parser.collector import collect_profile_links from app.parser.collector import collect_profile_links
from app.parser.profile import parse_person_profile from app.parser.profile import parse_person_profile
from app.parser.profile_url import profile_key from app.parser.profile_url import profile_key
@@ -68,7 +68,7 @@ def run_crawl(db: Session, settings: Settings) -> CrawlRun:
finally: finally:
time.sleep(settings.request_delay_seconds) time.sleep(settings.request_delay_seconds)
run.dismissed_count = _mark_dismissed(db, found_keys) run.dismissed_count = _mark_dismissed(db, run, found_keys, session, settings.request_timeout)
run.status = "completed" run.status = "completed"
except Exception as exc: except Exception as exc:
run.status = "failed" run.status = "failed"
@@ -107,6 +107,9 @@ def _upsert_employee(db: Session, run: CrawlRun, parsed: dict) -> Employee:
) )
db.add(employee) db.add(employee)
run.new_count += 1 run.new_count += 1
is_new = True
else:
is_new = False
employee.full_name = parsed.get("full_name") employee.full_name = parsed.get("full_name")
employee.status = "active" employee.status = "active"
@@ -117,6 +120,16 @@ def _upsert_employee(db: Session, run: CrawlRun, parsed: dict) -> Employee:
employee.current_checksum = checksum employee.current_checksum = checksum
db.flush() db.flush()
if is_new:
_record_employee_change(
db,
run,
employee,
"new",
profile_available=True,
message="Сотрудник впервые найден в источнике.",
)
db.query(ProfileTab).filter(ProfileTab.employee_id == employee.id).delete() db.query(ProfileTab).filter(ProfileTab.employee_id == employee.id).delete()
for tab in parsed.get("tabs") or []: for tab in parsed.get("tabs") or []:
db.add( db.add(
@@ -141,20 +154,70 @@ def _upsert_employee(db: Session, run: CrawlRun, parsed: dict) -> Employee:
return employee return employee
def _mark_dismissed(db: Session, found_keys: set[str]) -> int: def _mark_dismissed(db: Session, run: CrawlRun, found_keys: set[str], session: requests.Session, timeout: int) -> int:
dismissed = 0 dismissed = 0
active = db.scalars(select(Employee).where(Employee.status == "active")).all() active = db.scalars(select(Employee).where(Employee.status == "active")).all()
now = datetime.now(timezone.utc) now = datetime.now(timezone.utc)
for employee in active: for employee in active:
if employee.profile_key in found_keys: if employee.profile_key in found_keys:
continue continue
profile_available = _profile_is_available(session, employee.canonical_url, timeout)
if profile_available:
_record_employee_change(
db,
run,
employee,
"missing_from_source",
profile_available=True,
message="Профиль доступен, но ссылка отсутствует в исходном списке.",
)
continue
employee.status = "dismissed" employee.status = "dismissed"
employee.dismissed_at = now employee.dismissed_at = now
_record_employee_change(
db,
run,
employee,
"dismissed",
profile_available=False,
message="Сотрудник отсутствует в исходном списке, профиль не подтвердился как доступный.",
)
dismissed += 1 dismissed += 1
db.commit() db.commit()
return dismissed return dismissed
def _profile_is_available(session: requests.Session, url: str, timeout: int) -> bool:
try:
response = session.get(url, headers=HEADERS, timeout=timeout, allow_redirects=True)
return response.status_code < 400
except requests.RequestException:
return False
def _record_employee_change(
db: Session,
run: CrawlRun,
employee: Employee,
change_type: str,
*,
profile_available: bool | None,
message: str,
) -> None:
db.add(
CrawlRunEmployeeChange(
crawl_run_id=run.id,
employee_id=employee.id,
profile_key=employee.profile_key,
profile_url=employee.canonical_url,
full_name=employee.full_name,
change_type=change_type,
profile_available=profile_available,
message=message,
)
)
def _checksum(data: dict) -> str: def _checksum(data: dict) -> str:
payload = json.dumps(data, ensure_ascii=False, sort_keys=True, separators=(",", ":")) payload = json.dumps(data, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
return hashlib.sha256(payload.encode("utf-8")).hexdigest() return hashlib.sha256(payload.encode("utf-8")).hexdigest()

View File

@@ -1,6 +1,8 @@
.admin { .admin {
margin: 0; margin: 0;
min-height: 100vh; min-height: 100vh;
display: flex;
flex-direction: column;
color: #1f2937; color: #1f2937;
background: #f6f7f9; background: #f6f7f9;
font-family: Arial, sans-serif; font-family: Arial, sans-serif;
@@ -21,6 +23,11 @@
font-size: 20px; font-size: 20px;
} }
.admin__brand-link {
color: inherit;
text-decoration: none;
}
.admin__nav { .admin__nav {
display: flex; display: flex;
align-items: center; align-items: center;
@@ -34,6 +41,7 @@
} }
.admin__main { .admin__main {
flex: 1;
width: min(1180px, calc(100% - 32px)); width: min(1180px, calc(100% - 32px));
margin: 28px auto; margin: 28px auto;
} }
@@ -52,18 +60,30 @@
} }
.metric { .metric {
display: block;
padding: 18px; padding: 18px;
background: #ffffff; background: #ffffff;
border: 1px solid #d9dee7; border: 1px solid #d9dee7;
border-radius: 8px; border-radius: 8px;
} }
.metric--link {
color: inherit;
text-decoration: none;
}
.metric--link:hover {
border-color: #0f766e;
}
.metric__label { .metric__label {
display: block;
color: #6b7280; color: #6b7280;
font-size: 13px; font-size: 13px;
} }
.metric__value { .metric__value {
display: block;
margin-top: 8px; margin-top: 8px;
font-size: 28px; font-size: 28px;
font-weight: 700; font-weight: 700;
@@ -87,6 +107,14 @@
border-collapse: collapse; border-collapse: collapse;
} }
.table__row {
cursor: pointer;
}
.table__row:hover {
background: #f0fdfa;
}
.table__cell, .table__cell,
.table__head { .table__head {
padding: 10px 8px; padding: 10px 8px;
@@ -270,6 +298,18 @@
line-height: 1.55; line-height: 1.55;
} }
.employee-section__meta {
display: flex;
flex-wrap: wrap;
gap: 8px 12px;
color: #4b5563;
font-size: 13px;
}
.employee-section__meta-item {
line-height: 1.4;
}
.employee-section__table-wrap { .employee-section__table-wrap {
overflow-x: auto; overflow-x: auto;
} }
@@ -319,12 +359,22 @@
} }
.stats-strip__item { .stats-strip__item {
display: block;
padding: 14px 16px; padding: 14px 16px;
background: #ffffff; background: #ffffff;
border: 1px solid #d9dee7; border: 1px solid #d9dee7;
border-radius: 8px; border-radius: 8px;
} }
.stats-strip__item--link {
color: inherit;
text-decoration: none;
}
.stats-strip__item--link:hover {
border-color: #0f766e;
}
.stats-strip__label { .stats-strip__label {
display: block; display: block;
color: #6b7280; color: #6b7280;

View File

@@ -79,7 +79,7 @@
const errors = document.querySelector("[data-progress-errors]"); const errors = document.querySelector("[data-progress-errors]");
const fill = document.querySelector("[data-progress-fill]"); const fill = document.querySelector("[data-progress-fill]");
const percent = document.querySelector("[data-progress-percent]"); const percent = document.querySelector("[data-progress-percent]");
if (status) status.textContent = run.status; if (status) status.textContent = run.status_display || run.status;
if (processed) processed.textContent = run.processed_count; if (processed) processed.textContent = run.processed_count;
if (found) found.textContent = run.found_count; if (found) found.textContent = run.found_count;
if (errors) errors.textContent = run.error_count; if (errors) errors.textContent = run.error_count;

View File

@@ -8,14 +8,13 @@
</head> </head>
<body class="admin"> <body class="admin">
<header class="admin__header"> <header class="admin__header">
<h1 class="admin__brand">MIEM Employees</h1> <h1 class="admin__brand"><a class="admin__brand-link" href="/admin">MIEM Employees</a></h1>
<nav class="admin__nav"> <nav class="admin__nav">
<a class="admin__link" href="/admin">Dashboard</a> <a class="admin__link" href="/admin">Обзор</a>
<a class="admin__link" href="/admin/directory">Directory</a> <a class="admin__link" href="/admin/directory">Сотрудники</a>
<a class="admin__link" href="/admin/employees">Employees</a> <a class="admin__link" href="/admin/runs">Запуски</a>
<a class="admin__link" href="/admin/runs">Runs</a>
<form method="post" action="/admin/logout"> <form method="post" action="/admin/logout">
<button class="button button--ghost" type="submit">Logout</button> <button class="button button--ghost" type="submit">Выйти</button>
</form> </form>
</nav> </nav>
</header> </header>

View File

@@ -1,43 +1,43 @@
{% extends "base.html" %} {% extends "base.html" %}
{% block title %}Dashboard · MIEM Employees{% endblock %} {% block title %}Обзор · MIEM Employees{% endblock %}
{% block content %} {% block content %}
<section class="admin__grid"> <section class="admin__grid">
<div class="metric"><div class="metric__label">Total</div><div class="metric__value">{{ counts.total }}</div></div> <a class="metric metric--link" href="/admin/directory"><span class="metric__label">Всего в базе</span><span class="metric__value">{{ counts.total }}</span></a>
<div class="metric"><div class="metric__label">Active</div><div class="metric__value">{{ counts.active }}</div></div> <a class="metric metric--link" href="/admin/directory?status=active"><span class="metric__label">Работают</span><span class="metric__value">{{ counts.active }}</span></a>
<div class="metric"><div class="metric__label">New in last run</div><div class="metric__value">{{ counts.new_in_last_run }}</div></div> <a class="metric metric--link" href="{% if latest_run %}/admin/runs/{{ latest_run.id }}#new-employees{% else %}/admin/runs{% endif %}"><span class="metric__label">Новые за запуск</span><span class="metric__value">{{ counts.new_in_last_run }}</span></a>
<div class="metric"><div class="metric__label">Dismissed</div><div class="metric__value">{{ counts.dismissed }}</div></div> <a class="metric metric--link" href="/admin/directory?status=dismissed"><span class="metric__label">Уволены</span><span class="metric__value">{{ counts.dismissed }}</span></a>
</section> </section>
<section class="stats-strip"> <section class="stats-strip">
<div class="stats-strip__item"> <div class="stats-strip__item">
<span class="stats-strip__label">Latest added</span> <span class="stats-strip__label">Последний добавленный</span>
{% if counts.latest_added %} {% if counts.latest_added %}
<a class="stats-strip__value" href="/admin/employees/{{ counts.latest_added.id }}">{{ counts.latest_added.full_name or counts.latest_added.canonical_url }}</a> <a class="stats-strip__value" href="/admin/employees/{{ counts.latest_added.id }}">{{ counts.latest_added.full_name or counts.latest_added.canonical_url }}</a>
{% else %} {% else %}
<span class="stats-strip__value">No employees yet</span> <span class="stats-strip__value">Сотрудников пока нет</span>
{% endif %} {% endif %}
</div> </div>
<div class="stats-strip__item"> <a class="stats-strip__item stats-strip__item--link" href="/admin/runs">
<span class="stats-strip__label">Runs</span> <span class="stats-strip__label">Запуски</span>
<span class="stats-strip__value">{{ counts.runs }}</span> <span class="stats-strip__value">{{ counts.runs }}</span>
</div> </a>
<div class="stats-strip__item"> <div class="stats-strip__item">
<span class="stats-strip__label">Errors</span> <span class="stats-strip__label">Ошибки</span>
<span class="stats-strip__value">{{ counts.errors }}</span> <span class="stats-strip__value">{{ counts.errors }}</span>
</div> </div>
</section> </section>
<section class="panel progress-panel" data-progress-panel> <section class="panel progress-panel" data-progress-panel>
<div class="progress-panel__header"> <div class="progress-panel__header">
<h2 class="panel__title">Parsing progress</h2> <h2 class="panel__title">Прогресс парсинга</h2>
<form method="post" action="/admin/crawl-now"> <form method="post" action="/admin/crawl-now">
<button class="button" type="submit">Start crawl now</button> <button class="button" type="submit">Запустить парсинг</button>
</form> </form>
</div> </div>
{% set run = counts.current_running_run or latest_run %} {% set run = counts.current_running_run or latest_run %}
<div class="progress-panel__body" data-progress-body> <div class="progress-panel__body" data-progress-body>
<div class="progress-panel__meta"> <div class="progress-panel__meta">
<span data-progress-status>{{ run.status if run else "idle" }}</span> <span data-progress-status>{{ run.status_display if run else "Ожидание" }}</span>
<span><span data-progress-processed>{{ run.processed_count if run else 0 }}</span> / <span data-progress-found>{{ run.found_count if run else 0 }}</span> processed</span> <span>обработано: <span data-progress-processed>{{ run.processed_count if run else 0 }}</span> / <span data-progress-found>{{ run.found_count if run else 0 }}</span></span>
<span><span data-progress-errors>{{ run.error_count if run else 0 }}</span> errors</span> <span>ошибок: <span data-progress-errors>{{ run.error_count if run else 0 }}</span></span>
</div> </div>
<div class="progress-bar" aria-label="Parsing progress"> <div class="progress-bar" aria-label="Parsing progress">
<div class="progress-bar__fill" data-progress-fill style="width: {{ run.progress_percent if run else 0 }}%"></div> <div class="progress-bar__fill" data-progress-fill style="width: {{ run.progress_percent if run else 0 }}%"></div>
@@ -46,12 +46,12 @@
</div> </div>
</section> </section>
<section class="panel"> <section class="panel">
<h2 class="panel__title">Latest runs</h2> <h2 class="panel__title">Последние запуски</h2>
<table class="table"> <table class="table">
<thead><tr><th class="table__head">ID</th><th class="table__head">Status</th><th class="table__head">Parsed</th><th class="table__head">Errors</th><th class="table__head">Started</th></tr></thead> <thead><tr><th class="table__head">ID</th><th class="table__head">Статус</th><th class="table__head">Обработано</th><th class="table__head">Ошибки</th><th class="table__head">Старт</th></tr></thead>
<tbody> <tbody>
{% for run in runs %} {% for run in runs %}
<tr><td class="table__cell">{{ run.id }}</td><td class="table__cell">{{ run.status }}</td><td class="table__cell">{{ run.parsed_count }}</td><td class="table__cell">{{ run.error_count }}</td><td class="table__cell">{{ run.started_at }}</td></tr> <tr><td class="table__cell">{{ run.id }}</td><td class="table__cell">{{ run.status_display }}</td><td class="table__cell">{{ run.parsed_count }}</td><td class="table__cell">{{ run.error_count }}</td><td class="table__cell">{{ run.started_display }}</td></tr>
{% endfor %} {% endfor %}
</tbody> </tbody>
</table> </table>

View File

@@ -1,65 +1,71 @@
{% extends "base.html" %} {% extends "base.html" %}
{% block title %}Directory · MIEM Employees{% endblock %} {% block title %}Сотрудники · MIEM Employees{% endblock %}
{% block content %} {% block content %}
<section class="directory"> <section class="directory">
<div class="directory__header"> <div class="directory__header">
<div> <div>
<h2 class="directory__title">Directory</h2> <h2 class="directory__title">Сотрудники</h2>
<p class="directory__summary">{{ page.total }} employees found</p> <p class="directory__summary">Найдено: {{ page.total }}</p>
</div> </div>
<button class="button" type="button" data-columns-open>Columns</button> <button class="button" type="button" data-columns-open>Колонки</button>
</div> </div>
<form class="directory__filters" method="get" action="/admin/directory"> <form class="directory__filters" method="get" action="/admin/directory">
<input class="directory__input" name="q" value="{{ filters.q }}" placeholder="Name or URL"> <input class="directory__input" name="q" value="{{ filters.q }}" placeholder="ФИО или ссылка">
<select class="directory__input" name="status"> <select class="directory__input" name="status">
<option value="" {% if not filters.status %}selected{% endif %}>All statuses</option> <option value="" {% if not filters.status %}selected{% endif %}>Все статусы</option>
<option value="active" {% if filters.status == "active" %}selected{% endif %}>Active</option> <option value="active" {% if filters.status == "active" %}selected{% endif %}>Работает</option>
<option value="dismissed" {% if filters.status == "dismissed" %}selected{% endif %}>Dismissed</option> <option value="dismissed" {% if filters.status == "dismissed" %}selected{% endif %}>Уволен</option>
</select> </select>
<select class="directory__input" name="has_email"> <select class="directory__input" name="has_email">
<option value="" {% if not filters.has_email %}selected{% endif %}>Any email</option> <option value="" {% if not filters.has_email %}selected{% endif %}>Любой email</option>
<option value="true" {% if filters.has_email == "true" %}selected{% endif %}>Has email</option> <option value="true" {% if filters.has_email == "true" %}selected{% endif %}>Есть email</option>
<option value="false" {% if filters.has_email == "false" %}selected{% endif %}>No email</option> <option value="false" {% if filters.has_email == "false" %}selected{% endif %}>Нет email</option>
</select> </select>
<input class="directory__input" type="date" name="started_from" value="{{ filters.started_from }}" aria-label="First seen from"> <input class="directory__input" type="date" name="started_from" value="{{ filters.started_from }}" aria-label="Впервые найден с">
<input class="directory__input" type="date" name="started_to" value="{{ filters.started_to }}" aria-label="First seen to"> <input class="directory__input" type="date" name="started_to" value="{{ filters.started_to }}" aria-label="Впервые найден по">
<select class="directory__input" name="sort"> <select class="directory__input" name="sort">
{% for value, label in [("full_name", "Name"), ("status", "Status"), ("hse_start_year", "HSE start"), ("first_seen_at", "First seen"), ("last_seen_at", "Last seen"), ("dismissed_at", "Dismissed")] %} {% for value, label in [("full_name", "ФИО"), ("status", "Статус"), ("hse_start_year", "Год начала"), ("first_seen_at", "Впервые найден"), ("last_seen_at", "Последний раз найден"), ("dismissed_at", "Дата увольнения")] %}
<option value="{{ value }}" {% if filters.sort == value %}selected{% endif %}>Sort: {{ label }}</option> <option value="{{ value }}" {% if filters.sort == value %}selected{% endif %}>Сортировка: {{ label }}</option>
{% endfor %} {% endfor %}
</select> </select>
<select class="directory__input" name="direction"> <select class="directory__input" name="direction">
<option value="asc" {% if filters.direction == "asc" %}selected{% endif %}>Ascending</option> <option value="asc" {% if filters.direction == "asc" %}selected{% endif %}>По возрастанию</option>
<option value="desc" {% if filters.direction == "desc" %}selected{% endif %}>Descending</option> <option value="desc" {% if filters.direction == "desc" %}selected{% endif %}>По убыванию</option>
</select> </select>
<button class="button" type="submit">Apply</button> <select class="directory__input" name="limit" onchange="this.form.offset.value = 0; this.form.submit()">
{% for value in [25, 50, 100] %}
<option value="{{ value }}" {% if filters.limit == value %}selected{% endif %}>На странице: {{ value }}</option>
{% endfor %}
</select>
<input type="hidden" name="offset" value="{{ filters.offset }}">
<button class="button" type="submit">Применить</button>
</form> </form>
<div class="directory__table-wrap"> <div class="directory__table-wrap">
<table class="directory-table" data-directory-table> <table class="directory-table" data-directory-table>
<thead> <thead>
<tr> <tr>
<th class="directory-table__head" data-column="full_name">Name</th> <th class="directory-table__head" data-column="full_name">ФИО</th>
<th class="directory-table__head" data-column="status">Status</th> <th class="directory-table__head" data-column="status">Статус</th>
<th class="directory-table__head" data-column="positions">Positions</th> <th class="directory-table__head" data-column="positions">Должности</th>
<th class="directory-table__head" data-column="hse_start_year">HSE start</th> <th class="directory-table__head" data-column="hse_start_year">Год начала</th>
<th class="directory-table__head" data-column="email">Email</th> <th class="directory-table__head" data-column="email">Email</th>
<th class="directory-table__head" data-column="phone">Phone</th> <th class="directory-table__head" data-column="phone">Телефон</th>
<th class="directory-table__head" data-column="address">Address</th> <th class="directory-table__head" data-column="address">Адрес</th>
<th class="directory-table__head" data-column="publications_count">Publications</th> <th class="directory-table__head" data-column="publications_count">Публикации</th>
<th class="directory-table__head" data-column="courses_count">Courses</th> <th class="directory-table__head" data-column="courses_count">Курсы</th>
<th class="directory-table__head" data-column="first_seen_at">First seen</th> <th class="directory-table__head" data-column="first_seen_at">Впервые найден</th>
<th class="directory-table__head" data-column="last_seen_at">Last seen</th> <th class="directory-table__head" data-column="last_seen_at">Последний раз найден</th>
<th class="directory-table__head" data-column="dismissed_at">Dismissed</th> <th class="directory-table__head" data-column="dismissed_at">Дата увольнения</th>
<th class="directory-table__head" data-column="profile">Profile</th> <th class="directory-table__head" data-column="profile">Профиль</th>
</tr> </tr>
</thead> </thead>
<tbody> <tbody>
{% for employee in page.items %} {% for employee in page.employees %}
<tr class="directory-table__row" data-row-href="/admin/employees/{{ employee.id }}"> <tr class="directory-table__row" data-row-href="/admin/employees/{{ employee.id }}">
<td class="directory-table__cell" data-column="full_name">{{ employee.full_name or "No name" }}</td> <td class="directory-table__cell" data-column="full_name">{{ employee.full_name or "Без имени" }}</td>
<td class="directory-table__cell" data-column="status"><span class="badge {% if employee.status == "dismissed" %}badge--dismissed{% endif %}">{{ employee.status }}</span></td> <td class="directory-table__cell" data-column="status"><span class="badge {% if employee.status == "dismissed" %}badge--dismissed{% endif %}">{{ employee.status_display }}</span></td>
<td class="directory-table__cell" data-column="positions">{{ employee.positions_text }}</td> <td class="directory-table__cell" data-column="positions">{{ employee.positions_text }}</td>
<td class="directory-table__cell" data-column="hse_start_year">{{ employee.hse_start_year or "" }}</td> <td class="directory-table__cell" data-column="hse_start_year">{{ employee.hse_start_year or "" }}</td>
<td class="directory-table__cell" data-column="email">{{ employee.email_text }}</td> <td class="directory-table__cell" data-column="email">{{ employee.email_text }}</td>
@@ -67,13 +73,13 @@
<td class="directory-table__cell" data-column="address">{{ employee.address or "" }}</td> <td class="directory-table__cell" data-column="address">{{ employee.address or "" }}</td>
<td class="directory-table__cell" data-column="publications_count">{{ employee.publications_count }}</td> <td class="directory-table__cell" data-column="publications_count">{{ employee.publications_count }}</td>
<td class="directory-table__cell" data-column="courses_count">{{ employee.courses_count }}</td> <td class="directory-table__cell" data-column="courses_count">{{ employee.courses_count }}</td>
<td class="directory-table__cell" data-column="first_seen_at">{{ employee.first_seen_at or "" }}</td> <td class="directory-table__cell" data-column="first_seen_at">{{ employee.first_seen_display }}</td>
<td class="directory-table__cell" data-column="last_seen_at">{{ employee.last_seen_at or "" }}</td> <td class="directory-table__cell" data-column="last_seen_at">{{ employee.last_seen_display }}</td>
<td class="directory-table__cell" data-column="dismissed_at">{{ employee.dismissed_at or "" }}</td> <td class="directory-table__cell" data-column="dismissed_at">{{ employee.dismissed_display }}</td>
<td class="directory-table__cell" data-column="profile"><a class="admin__link" href="{{ employee.canonical_url }}">Open</a></td> <td class="directory-table__cell" data-column="profile"><a class="admin__link" href="{{ employee.canonical_url }}">Открыть</a></td>
</tr> </tr>
{% else %} {% else %}
<tr><td class="directory-table__empty" colspan="13">No employees match these filters.</td></tr> <tr><td class="directory-table__empty" colspan="13">По этим фильтрам сотрудники не найдены.</td></tr>
{% endfor %} {% endfor %}
</tbody> </tbody>
</table> </table>
@@ -83,24 +89,24 @@
{% set prev_offset = filters.offset - filters.limit %} {% set prev_offset = filters.offset - filters.limit %}
{% set next_offset = filters.offset + filters.limit %} {% set next_offset = filters.offset + filters.limit %}
{% if filters.offset > 0 %} {% if filters.offset > 0 %}
<a class="admin__link" href="{{ request.url.include_query_params(offset=prev_offset) }}">Previous</a> <a class="admin__link" href="{{ request.url.include_query_params(offset=prev_offset) }}">Назад</a>
{% endif %} {% endif %}
<span class="directory__page">Page {{ page.page }}{% if page.pages %} of {{ page.pages }}{% endif %}</span> <span class="directory__page">Страница {{ page.page }}{% if page.pages %} из {{ page.pages }}{% endif %}</span>
{% if next_offset < page.total %} {% if next_offset < page.total %}
<a class="admin__link" href="{{ request.url.include_query_params(offset=next_offset) }}">Next</a> <a class="admin__link" href="{{ request.url.include_query_params(offset=next_offset) }}">Вперед</a>
{% endif %} {% endif %}
</div> </div>
</section> </section>
<div class="columns-modal" data-columns-modal hidden> <div class="columns-modal" data-columns-modal hidden>
<div class="columns-modal__backdrop" data-columns-close></div> <div class="columns-modal__backdrop" data-columns-close></div>
<section class="columns-modal__panel" aria-label="Column settings"> <section class="columns-modal__panel" aria-label="Настройка колонок">
<div class="columns-modal__header"> <div class="columns-modal__header">
<h3 class="columns-modal__title">Visible columns</h3> <h3 class="columns-modal__title">Отображаемые колонки</h3>
<button class="button button--ghost" type="button" data-columns-close>Close</button> <button class="button button--ghost" type="button" data-columns-close>Закрыть</button>
</div> </div>
<div class="columns-modal__grid"> <div class="columns-modal__grid">
{% for key, label in [("full_name", "Name"), ("status", "Status"), ("positions", "Positions"), ("hse_start_year", "HSE start"), ("email", "Email"), ("phone", "Phone"), ("address", "Address"), ("publications_count", "Publications"), ("courses_count", "Courses"), ("first_seen_at", "First seen"), ("last_seen_at", "Last seen"), ("dismissed_at", "Dismissed"), ("profile", "Profile")] %} {% for key, label in [("full_name", "ФИО"), ("status", "Статус"), ("positions", "Должности"), ("hse_start_year", "Год начала"), ("email", "Email"), ("phone", "Телефон"), ("address", "Адрес"), ("publications_count", "Публикации"), ("courses_count", "Курсы"), ("first_seen_at", "Впервые найден"), ("last_seen_at", "Последний раз найден"), ("dismissed_at", "Дата увольнения"), ("profile", "Профиль")] %}
<label class="columns-modal__option"><input class="columns-modal__checkbox" type="checkbox" value="{{ key }}" data-column-toggle> {{ label }}</label> <label class="columns-modal__option"><input class="columns-modal__checkbox" type="checkbox" value="{{ key }}" data-column-toggle> {{ label }}</label>
{% endfor %} {% endfor %}
</div> </div>

View File

@@ -5,7 +5,7 @@
<div class="employee-card__header"> <div class="employee-card__header">
<div class="employee-card__identity"> <div class="employee-card__identity">
<h2 class="employee-card__title">{{ employee_view.full_name or employee.profile_key }}</h2> <h2 class="employee-card__title">{{ employee_view.full_name or employee.profile_key }}</h2>
<span class="badge {% if employee_view.status == "dismissed" %}badge--dismissed{% endif %}">{{ employee_view.status }}</span> <span class="badge {% if employee_view.status == "dismissed" %}badge--dismissed{% endif %}">{{ employee_view.status_display }}</span>
</div> </div>
<a class="admin__link" href="{{ employee_view.canonical_url }}">{{ employee_view.canonical_url }}</a> <a class="admin__link" href="{{ employee_view.canonical_url }}">{{ employee_view.canonical_url }}</a>
</div> </div>
@@ -28,12 +28,11 @@
</dd> </dd>
</div> </div>
<div class="employee-card__meta-item"><dt class="employee-card__meta-label">Год начала работы в ВШЭ</dt><dd class="employee-card__meta-value">{{ employee_view.hse_start_year or "Не указано" }}</dd></div> <div class="employee-card__meta-item"><dt class="employee-card__meta-label">Год начала работы в ВШЭ</dt><dd class="employee-card__meta-value">{{ employee_view.hse_start_year or "Не указано" }}</dd></div>
<div class="employee-card__meta-item"><dt class="employee-card__meta-label">Profile type</dt><dd class="employee-card__meta-value">{{ employee_view.profile_type or "Не указано" }}</dd></div> <div class="employee-card__meta-item"><dt class="employee-card__meta-label">Тип профиля</dt><dd class="employee-card__meta-value">{{ employee_view.profile_type or "Не указано" }}</dd></div>
<div class="employee-card__meta-item"><dt class="employee-card__meta-label">Profile ID</dt><dd class="employee-card__meta-value">{{ employee_view.profile_id or "Не указано" }}</dd></div> <div class="employee-card__meta-item"><dt class="employee-card__meta-label">ID профиля</dt><dd class="employee-card__meta-value">{{ employee_view.profile_id or "Не указано" }}</dd></div>
<div class="employee-card__meta-item"><dt class="employee-card__meta-label">First seen</dt><dd class="employee-card__meta-value">{{ employee_view.first_seen_at or "Не указано" }}</dd></div> <div class="employee-card__meta-item"><dt class="employee-card__meta-label">Впервые найден</dt><dd class="employee-card__meta-value">{{ employee_view.first_seen_display }}</dd></div>
<div class="employee-card__meta-item"><dt class="employee-card__meta-label">Last seen</dt><dd class="employee-card__meta-value">{{ employee_view.last_seen_at or "Не указано" }}</dd></div> <div class="employee-card__meta-item"><dt class="employee-card__meta-label">Последний раз найден</dt><dd class="employee-card__meta-value">{{ employee_view.last_seen_display }}</dd></div>
<div class="employee-card__meta-item"><dt class="employee-card__meta-label">Dismissed at</dt><dd class="employee-card__meta-value">{{ employee_view.dismissed_at or "Не указано" }}</dd></div> <div class="employee-card__meta-item"><dt class="employee-card__meta-label">Дата увольнения</dt><dd class="employee-card__meta-value">{{ employee_view.dismissed_display }}</dd></div>
<div class="employee-card__meta-item"><dt class="employee-card__meta-label">Parser version</dt><dd class="employee-card__meta-value">{{ employee_view.parser_version or "Не указано" }}</dd></div>
</dl> </dl>
</section> </section>
@@ -139,6 +138,34 @@
</li> </li>
{% endfor %} {% endfor %}
</ul> </ul>
{% elif section.type == "graduation_theses" and section.theses %}
{% if section.theses_count %}<p class="employee-section__note">Всего: {{ section.theses_count }}</p>{% endif %}
<ul class="employee-card__list">
{% for thesis in section.theses %}
<li class="employee-card__list-item">
{% if thesis.student %}<strong>{{ thesis.student }}</strong>{% endif %}
{% if thesis.title %}
<div class="employee-section__text">
{% if thesis.project_url %}
<a class="admin__link" href="{{ thesis.project_url }}">{{ thesis.title }}</a>
{% else %}
{{ thesis.title }}
{% endif %}
</div>
{% endif %}
<div class="employee-section__meta">
{% if thesis.defense_year %}<span class="employee-section__meta-item">Год защиты: {{ thesis.defense_year }}</span>{% endif %}
{% if thesis.level %}<span class="employee-section__meta-item">{{ thesis.level }}</span>{% endif %}
{% if thesis.rating is not none %}<span class="employee-section__meta-item">Оценка: {{ thesis.rating }}</span>{% endif %}
{% if thesis.program %}
<span class="employee-section__meta-item">
{% if thesis.program_url %}<a class="admin__link" href="{{ thesis.program_url }}">{{ thesis.program }}</a>{% else %}{{ thesis.program }}{% endif %}
</span>
{% endif %}
</div>
</li>
{% endfor %}
</ul>
{% elif section.type == "table" and section.table %} {% elif section.type == "table" and section.table %}
<div class="employee-section__table-wrap"> <div class="employee-section__table-wrap">
<table class="employee-section__table"> <table class="employee-section__table">
@@ -162,16 +189,16 @@
<p class="employee-section__text">{{ paragraph }}</p> <p class="employee-section__text">{{ paragraph }}</p>
{% endfor %} {% endfor %}
{% endif %} {% endif %}
{% if section.items %} {% if section.list_items %}
<ul class="employee-card__list"> <ul class="employee-card__list">
{% for item in section.items %} {% for item in section.list_items %}
<li class="employee-card__list-item">{{ item }}</li> <li class="employee-card__list-item">{{ item }}</li>
{% endfor %} {% endfor %}
</ul> </ul>
{% endif %} {% endif %}
{% endif %} {% endif %}
{% if section.links and section.type not in ["courses_by_year"] %} {% if section.links and section.type not in ["courses_by_year", "graduation_theses"] %}
<div class="employee-section__links"> <div class="employee-section__links">
{% for link in section.links %} {% for link in section.links %}
<a class="employee-section__link" href="{{ link.url }}">{{ link.text }}</a> <a class="employee-section__link" href="{{ link.url }}">{{ link.text }}</a>
@@ -188,12 +215,12 @@
</section> </section>
<section class="panel"> <section class="panel">
<h2 class="panel__title">Snapshots</h2> <h2 class="panel__title">Снапшоты</h2>
<table class="table"> <table class="table">
<thead><tr><th class="table__head">Captured</th><th class="table__head">Checksum</th><th class="table__head">Parser</th></tr></thead> <thead><tr><th class="table__head">Дата</th><th class="table__head">Checksum</th><th class="table__head">Парсер</th></tr></thead>
<tbody> <tbody>
{% for snapshot in snapshots %} {% for snapshot in snapshots %}
<tr><td class="table__cell">{{ snapshot.captured_at }}</td><td class="table__cell">{{ snapshot.checksum }}</td><td class="table__cell">{{ snapshot.parser_version }}</td></tr> <tr><td class="table__cell">{{ snapshot.captured_display }}</td><td class="table__cell">{{ snapshot.checksum }}</td><td class="table__cell">{{ snapshot.parser_version }}</td></tr>
{% endfor %} {% endfor %}
</tbody> </tbody>
</table> </table>

View File

@@ -1,29 +0,0 @@
{% extends "base.html" %}
{% block title %}Employees · MIEM Employees{% endblock %}
{% block content %}
<section class="panel">
<h2 class="panel__title">Employees</h2>
<form class="form" method="get" action="/admin/employees">
<input class="form__input" name="q" value="{{ q }}" placeholder="Name or URL">
<select class="form__select" name="status">
<option value="" {% if not status %}selected{% endif %}>All</option>
<option value="active" {% if status == "active" %}selected{% endif %}>Active</option>
<option value="dismissed" {% if status == "dismissed" %}selected{% endif %}>Dismissed</option>
</select>
<button class="button" type="submit">Search</button>
</form>
<table class="table">
<thead><tr><th class="table__head">Name</th><th class="table__head">Status</th><th class="table__head">Last seen</th><th class="table__head">Profile</th></tr></thead>
<tbody>
{% for employee in employees %}
<tr>
<td class="table__cell"><a class="admin__link" href="/admin/employees/{{ employee.id }}">{{ employee.full_name or employee.profile_key }}</a></td>
<td class="table__cell"><span class="badge {% if employee.status == "dismissed" %}badge--dismissed{% endif %}">{{ employee.status }}</span></td>
<td class="table__cell">{{ employee.last_seen_at }}</td>
<td class="table__cell"><a class="admin__link" href="{{ employee.canonical_url }}">{{ employee.canonical_url }}</a></td>
</tr>
{% endfor %}
</tbody>
</table>
</section>
{% endblock %}

View File

@@ -3,18 +3,18 @@
<head> <head>
<meta charset="utf-8"> <meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1"> <meta name="viewport" content="width=device-width, initial-scale=1">
<title>Login · MIEM Employees</title> <title>Вход · MIEM Employees</title>
<link rel="stylesheet" href="/static/admin.css"> <link rel="stylesheet" href="/static/admin.css">
</head> </head>
<body class="admin"> <body class="admin">
<main class="admin__main"> <main class="admin__main">
<section class="panel"> <section class="panel">
<h1 class="panel__title">Admin login</h1> <h1 class="panel__title">Вход в админку</h1>
{% if error %}<p>{{ error }}</p>{% endif %} {% if error %}<p>{{ error }}</p>{% endif %}
<form class="form" method="post" action="/admin/login"> <form class="form" method="post" action="/admin/login">
<label class="form__label">Login <input class="form__input" name="username" autocomplete="username"></label> <label class="form__label">Логин <input class="form__input" name="username" autocomplete="username"></label>
<label class="form__label">Password <input class="form__input" name="password" type="password" autocomplete="current-password"></label> <label class="form__label">Пароль <input class="form__input" name="password" type="password" autocomplete="current-password"></label>
<button class="button" type="submit">Sign in</button> <button class="button" type="submit">Войти</button>
</form> </form>
</section> </section>
</main> </main>

View File

@@ -0,0 +1,64 @@
{% extends "base.html" %}
{% block title %}Запуск {{ run.id }} · MIEM Employees{% endblock %}
{% block content %}
<section class="panel">
<div class="progress-panel__header">
<div>
<h2 class="panel__title">Запуск {{ run.id }}</h2>
<p class="progress-panel__empty">{{ run.started_display }} · {{ run.status_display }}</p>
</div>
<a class="admin__link" href="/admin/runs">Все запуски</a>
</div>
<div class="stats-strip">
<div class="stats-strip__item"><span class="stats-strip__label">Найдено</span><span class="stats-strip__value">{{ run.found_count }}</span></div>
<div class="stats-strip__item"><span class="stats-strip__label">Обработано</span><span class="stats-strip__value">{{ run.parsed_count }}</span></div>
<div class="stats-strip__item"><span class="stats-strip__label">Новые</span><span class="stats-strip__value">{{ run.new_count }}</span></div>
<div class="stats-strip__item"><span class="stats-strip__label">Потеряшки</span><span class="stats-strip__value">{{ run.changes.missing_from_source | length }}</span></div>
<div class="stats-strip__item"><span class="stats-strip__label">Уволены</span><span class="stats-strip__value">{{ run.dismissed_count }}</span></div>
<div class="stats-strip__item"><span class="stats-strip__label">Ошибки</span><span class="stats-strip__value">{{ run.error_count }}</span></div>
</div>
{% if not run.changes_detail_available %}
<p class="progress-panel__empty">Детализация сотрудников для этого запуска недоступна. Она сохраняется только для новых запусков после обновления.</p>
{% endif %}
</section>
{% for group, title in [("new", "Новые сотрудники"), ("missing_from_source", "Потеряшки"), ("dismissed", "Уволенные")] %}
<section class="panel"{% if group == "new" %} id="new-employees"{% endif %}>
<h2 class="panel__title">{{ title }}</h2>
{% set items = run.changes[group] %}
{% if items %}
<table class="table">
<thead><tr><th class="table__head">ФИО</th><th class="table__head">Профиль</th><th class="table__head">Проверка</th><th class="table__head">Комментарий</th></tr></thead>
<tbody>
{% for item in items %}
<tr>
<td class="table__cell">{% if item.employee_id %}<a class="admin__link" href="/admin/employees/{{ item.employee_id }}">{{ item.full_name or item.profile_key }}</a>{% else %}{{ item.full_name or item.profile_key }}{% endif %}</td>
<td class="table__cell"><a class="admin__link" href="{{ item.profile_url }}">{{ item.profile_url }}</a></td>
<td class="table__cell">{{ item.profile_available_display }}</td>
<td class="table__cell">{{ item.message or "" }}</td>
</tr>
{% endfor %}
</tbody>
</table>
{% else %}
<p class="progress-panel__empty">Нет записей.</p>
{% endif %}
</section>
{% endfor %}
<section class="panel">
<h2 class="panel__title">Ошибки запуска</h2>
{% if run.errors %}
<table class="table">
<thead><tr><th class="table__head">Профиль</th><th class="table__head">Ошибка</th><th class="table__head">Время</th></tr></thead>
<tbody>
{% for error in run.errors %}
<tr><td class="table__cell">{{ error.profile_url or "" }}</td><td class="table__cell">{{ error.error_type }}: {{ error.message }}</td><td class="table__cell">{{ error.created_display }}</td></tr>
{% endfor %}
</tbody>
</table>
{% else %}
<p class="progress-panel__empty">Ошибок нет.</p>
{% endif %}
</section>
{% endblock %}

View File

@@ -1,10 +1,10 @@
{% extends "base.html" %} {% extends "base.html" %}
{% block title %}Runs · MIEM Employees{% endblock %} {% block title %}Запуски · MIEM Employees{% endblock %}
{% block content %} {% block content %}
<section class="panel"> <section class="panel">
<div class="progress-panel__header"> <div class="progress-panel__header">
<h2 class="panel__title">Crawl runs</h2> <h2 class="panel__title">Запуски парсинга</h2>
<form method="post" action="/admin/runs"><button class="button" type="submit">Start crawl now</button></form> <form method="post" action="/admin/runs"><button class="button" type="submit">Запустить парсинг</button></form>
</div> </div>
{% set run = runs[0] if runs else none %} {% set run = runs[0] if runs else none %}
{% if run %} {% if run %}
@@ -12,9 +12,9 @@
{% set percent = ((processed / run.found_count) * 100) | round(1) if run.found_count else 0 %} {% set percent = ((processed / run.found_count) * 100) | round(1) if run.found_count else 0 %}
<div class="progress-panel" data-progress-panel> <div class="progress-panel" data-progress-panel>
<div class="progress-panel__meta"> <div class="progress-panel__meta">
<span data-progress-status>{{ run.status }}</span> <span data-progress-status>{{ run.status_display }}</span>
<span><span data-progress-processed>{{ processed }}</span> / <span data-progress-found>{{ run.found_count }}</span> processed</span> <span>обработано: <span data-progress-processed>{{ processed }}</span> / <span data-progress-found>{{ run.found_count }}</span></span>
<span><span data-progress-errors>{{ run.error_count }}</span> errors</span> <span>ошибок: <span data-progress-errors>{{ run.error_count }}</span></span>
</div> </div>
<div class="progress-bar" aria-label="Parsing progress"> <div class="progress-bar" aria-label="Parsing progress">
<div class="progress-bar__fill" data-progress-fill style="width: {{ percent }}%"></div> <div class="progress-bar__fill" data-progress-fill style="width: {{ percent }}%"></div>
@@ -24,9 +24,9 @@
{% else %} {% else %}
<div class="progress-panel" data-progress-panel> <div class="progress-panel" data-progress-panel>
<div class="progress-panel__meta"> <div class="progress-panel__meta">
<span data-progress-status>idle</span> <span data-progress-status>Ожидание</span>
<span><span data-progress-processed>0</span> / <span data-progress-found>0</span> processed</span> <span>обработано: <span data-progress-processed>0</span> / <span data-progress-found>0</span></span>
<span><span data-progress-errors>0</span> errors</span> <span>ошибок: <span data-progress-errors>0</span></span>
</div> </div>
<div class="progress-bar" aria-label="Parsing progress"> <div class="progress-bar" aria-label="Parsing progress">
<div class="progress-bar__fill" data-progress-fill style="width: 0%"></div> <div class="progress-bar__fill" data-progress-fill style="width: 0%"></div>
@@ -35,18 +35,18 @@
</div> </div>
{% endif %} {% endif %}
<table class="table"> <table class="table">
<thead><tr><th class="table__head">ID</th><th class="table__head">Status</th><th class="table__head">Found</th><th class="table__head">Parsed</th><th class="table__head">New</th><th class="table__head">Errors</th><th class="table__head">Dismissed</th></tr></thead> <thead><tr><th class="table__head">ID</th><th class="table__head">Статус</th><th class="table__head">Найдено</th><th class="table__head">Обработано</th><th class="table__head">Новые</th><th class="table__head">Ошибки</th><th class="table__head">Уволены</th><th class="table__head">Старт</th></tr></thead>
<tbody> <tbody>
{% for run in runs %} {% for run in runs %}
<tr><td class="table__cell">{{ run.id }}</td><td class="table__cell">{{ run.status }}</td><td class="table__cell">{{ run.found_count }}</td><td class="table__cell">{{ run.parsed_count }}</td><td class="table__cell">{{ run.new_count }}</td><td class="table__cell">{{ run.error_count }}</td><td class="table__cell">{{ run.dismissed_count }}</td></tr> <tr class="table__row" data-row-href="/admin/runs/{{ run.id }}"><td class="table__cell"><a class="admin__link" href="/admin/runs/{{ run.id }}">{{ run.id }}</a></td><td class="table__cell">{{ run.status_display }}</td><td class="table__cell">{{ run.found_count }}</td><td class="table__cell">{{ run.parsed_count }}</td><td class="table__cell">{{ run.new_count }}</td><td class="table__cell">{{ run.error_count }}</td><td class="table__cell">{{ run.dismissed_count }}</td><td class="table__cell">{{ run.started_display }}</td></tr>
{% endfor %} {% endfor %}
</tbody> </tbody>
</table> </table>
</section> </section>
<section class="panel"> <section class="panel">
<h2 class="panel__title">Recent errors</h2> <h2 class="panel__title">Последние ошибки</h2>
<table class="table"> <table class="table">
<thead><tr><th class="table__head">Run</th><th class="table__head">Profile</th><th class="table__head">Error</th></tr></thead> <thead><tr><th class="table__head">Запуск</th><th class="table__head">Профиль</th><th class="table__head">Ошибка</th></tr></thead>
<tbody> <tbody>
{% for error in errors %} {% for error in errors %}
<tr><td class="table__cell">{{ error.crawl_run_id }}</td><td class="table__cell">{{ error.profile_url }}</td><td class="table__cell">{{ error.error_type }}: {{ error.message }}</td></tr> <tr><td class="table__cell">{{ error.crawl_run_id }}</td><td class="table__cell">{{ error.profile_url }}</td><td class="table__cell">{{ error.error_type }}: {{ error.message }}</td></tr>

View File

@@ -1,3 +1,3 @@
APP_VERSION = "0.2.4" APP_VERSION = "0.4.1"
FRONTEND_VERSION = "0.2.4" FRONTEND_VERSION = "0.4.1"
BACKEND_VERSION = "0.2.4" BACKEND_VERSION = "0.4.1"

View File

@@ -47,5 +47,31 @@ services:
postgres: postgres:
condition: service_healthy condition: service_healthy
keycloak:
image: quay.io/keycloak/keycloak:latest
container_name: keycloak
restart: unless-stopped
environment:
KC_DB: postgres
KC_DB_URL: jdbc:postgresql://postgres:5432/${KEYCLOAK_DB_NAME}
KC_DB_USERNAME: ${KEYCLOAK_DB_USER}
KC_DB_PASSWORD: ${KEYCLOAK_DB_PASSWORD}
KEYCLOAK_ADMIN: ${KEYCLOAK_ADMIN}
KEYCLOAK_ADMIN_PASSWORD: ${KEYCLOAK_ADMIN_PASSWORD}
KC_HTTP_ENABLED: true
KC_PROXY_HEADERS: xforwarded
KC_HOSTNAME: ${KEYCLOAK_HOSTNAME}
KC_HEALTH_ENABLED: true
KC_METRICS_ENABLED: true
command: start
ports:
- "127.0.0.1:8080:8080"
depends_on:
postgres:
condition: service_healthy
volumes: volumes:
postgres_data: postgres_data:

View File

@@ -0,0 +1,21 @@
CREATE TABLE IF NOT EXISTS crawl_run_employee_changes (
id SERIAL PRIMARY KEY,
crawl_run_id INTEGER NOT NULL REFERENCES crawl_runs(id),
employee_id INTEGER REFERENCES employees(id),
profile_key VARCHAR(255) NOT NULL,
profile_url TEXT NOT NULL,
full_name TEXT,
change_type VARCHAR(32) NOT NULL,
profile_available BOOLEAN,
message TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS ix_crawl_run_employee_changes_run_id
ON crawl_run_employee_changes (crawl_run_id);
CREATE INDEX IF NOT EXISTS ix_crawl_run_employee_changes_employee_id
ON crawl_run_employee_changes (employee_id);
CREATE INDEX IF NOT EXISTS ix_crawl_run_employee_changes_change_type
ON crawl_run_employee_changes (change_type);

View File

@@ -1,6 +1,6 @@
[project] [project]
name = "miem-workers" name = "miem-workers"
version = "0.1.0" version = "0.4.0"
description = "MIEM employees parser, admin API, and MCP server" description = "MIEM employees parser, admin API, and MCP server"
requires-python = ">=3.11" requires-python = ">=3.11"
dependencies = [ dependencies = [
@@ -12,6 +12,7 @@ dependencies = [
"lxml>=5.2.0", "lxml>=5.2.0",
"psycopg[binary]>=3.2.0", "psycopg[binary]>=3.2.0",
"pydantic-settings>=2.4.0", "pydantic-settings>=2.4.0",
"PyJWT[crypto]>=2.9.0",
"python-multipart>=0.0.9", "python-multipart>=0.0.9",
"requests>=2.32.0", "requests>=2.32.0",
"sqlalchemy>=2.0.32", "sqlalchemy>=2.0.32",

View File

@@ -6,6 +6,7 @@ jinja2>=3.1.4
lxml>=5.2.0 lxml>=5.2.0
psycopg[binary]>=3.2.0 psycopg[binary]>=3.2.0
pydantic-settings>=2.4.0 pydantic-settings>=2.4.0
PyJWT[crypto]>=2.9.0
python-multipart>=0.0.9 python-multipart>=0.0.9
requests>=2.32.0 requests>=2.32.0
sqlalchemy>=2.0.32 sqlalchemy>=2.0.32

View File

@@ -1,15 +1,25 @@
from datetime import datetime, timezone from datetime import datetime, timezone
from app.models import CrawlRun, Employee from app.models import CrawlError, CrawlRun, CrawlRunEmployeeChange, Employee
from app.services.admin_data import ( from app.services.admin_data import (
employee_detail_payload, employee_detail_payload,
employee_display_payload, employee_display_payload,
format_admin_datetime,
list_employees_page, list_employees_page,
run_detail_payload,
run_payload, run_payload,
stats_payload, stats_payload,
) )
def test_format_admin_datetime_handles_datetime_string_and_none():
value = datetime(2026, 4, 28, 17, 13, 34, tzinfo=timezone.utc)
assert format_admin_datetime(value) == "28.04.2026 20:13"
assert format_admin_datetime("2026-04-28T17:13:34.448605+00:00") == "28.04.2026 20:13"
assert format_admin_datetime(None) == "Не указано"
def test_employee_display_payload_extracts_common_fields(db_session): def test_employee_display_payload_extracts_common_fields(db_session):
employee = Employee( employee = Employee(
profile_key="staff:person", profile_key="staff:person",
@@ -32,9 +42,11 @@ def test_employee_display_payload_extracts_common_fields(db_session):
payload = employee_display_payload(employee) payload = employee_display_payload(employee)
assert payload["positions_text"] == "Professor" assert payload["positions_text"] == "Professor"
assert payload["status_display"] == "Работает"
assert payload["email_text"] == "person@hse.ru" assert payload["email_text"] == "person@hse.ru"
assert payload["publications_count"] == 1 assert payload["publications_count"] == 1
assert payload["courses_count"] == 1 assert payload["courses_count"] == 1
assert payload["first_seen_display"] != "Не указано"
def test_employee_detail_payload_normalizes_human_readable_sections(db_session): def test_employee_detail_payload_normalizes_human_readable_sections(db_session):
@@ -74,6 +86,19 @@ def test_employee_detail_payload_normalizes_human_readable_sections(db_session):
"academic_year": "2025/2026", "academic_year": "2025/2026",
"courses": [{"title": "Course", "url": "https://example.test/course"}], "courses": [{"title": "Course", "url": "https://example.test/course"}],
}, },
{
"title": "ВКР",
"type": "graduation_theses",
"theses_count": 1,
"theses": [
{
"student": "Student Name",
"title": "Thesis title",
"defense_year": 2025,
"project_url": "https://www.hse.ru/edu/vkr/1",
}
],
},
{ {
"title": "Fallback", "title": "Fallback",
"type": "generic", "type": "generic",
@@ -91,7 +116,8 @@ def test_employee_detail_payload_normalizes_human_readable_sections(db_session):
assert payload["sections"][0]["year_entries"][0]["text"] == "Master degree" assert payload["sections"][0]["year_entries"][0]["text"] == "Master degree"
assert payload["sections"][1]["publications"][0]["title"] == "Paper" assert payload["sections"][1]["publications"][0]["title"] == "Paper"
assert payload["sections"][2]["courses"][0]["title"] == "Course" assert payload["sections"][2]["courses"][0]["title"] == "Course"
assert payload["sections"][3]["paragraphs"] == ["Fallback text"] assert payload["sections"][3]["theses"][0]["student"] == "Student Name"
assert payload["sections"][4]["paragraphs"] == ["Fallback text"]
def test_employee_payloads_tolerate_malformed_current_data(db_session): def test_employee_payloads_tolerate_malformed_current_data(db_session):
@@ -143,7 +169,8 @@ def test_list_employees_page_filters_sorts_and_paginates(db_session):
page = list_employees_page(db_session, status="active", sort="full_name", direction="asc", limit=10) page = list_employees_page(db_session, status="active", sort="full_name", direction="asc", limit=10)
assert page["total"] == 1 assert page["total"] == 1
assert page["items"][0]["full_name"] == "Alpha" assert page["employees"][0]["full_name"] == "Alpha"
assert page["limit"] == 50
def test_stats_payload_uses_latest_run_new_count(db_session): def test_stats_payload_uses_latest_run_new_count(db_session):
@@ -180,3 +207,44 @@ def test_run_payload_calculates_progress():
assert payload["processed_count"] == 5 assert payload["processed_count"] == 5
assert payload["progress_percent"] == 50.0 assert payload["progress_percent"] == 50.0
assert payload["status_display"] == "Выполняется"
def test_run_detail_payload_groups_changes_and_handles_old_runs(db_session):
old_run = CrawlRun(source_url="https://miem.hse.ru/persons", status="completed")
run = CrawlRun(source_url="https://miem.hse.ru/persons", status="completed", new_count=1)
employee = Employee(
profile_key="staff:new",
canonical_url="https://www.hse.ru/staff/new",
full_name="New Person",
status="active",
first_seen_at=datetime.now(timezone.utc),
last_seen_at=datetime.now(timezone.utc),
)
db_session.add_all([old_run, run, employee])
db_session.commit()
db_session.add(
CrawlRunEmployeeChange(
crawl_run_id=run.id,
employee_id=employee.id,
profile_key=employee.profile_key,
profile_url=employee.canonical_url,
full_name=employee.full_name,
change_type="new",
profile_available=True,
message="added",
)
)
db_session.add(
CrawlError(crawl_run_id=run.id, profile_url=employee.canonical_url, error_type="ValueError", message="bad")
)
db_session.commit()
payload = run_detail_payload(db_session, run)
old_payload = run_detail_payload(db_session, old_run)
assert payload["changes_detail_available"] is True
assert payload["changes"]["new"][0]["full_name"] == "New Person"
assert payload["errors"][0]["error_type"] == "ValueError"
assert old_payload["changes_detail_available"] is False
assert old_payload["changes"]["new"] == []

View File

@@ -0,0 +1,63 @@
from pathlib import Path
def test_base_navigation_is_russian_and_has_no_legacy_employees_link():
template = Path("app/templates/base.html").read_text(encoding="utf-8")
assert "Обзор" in template
assert "Сотрудники" in template
assert "Запуски" in template
assert "Выйти" in template
assert '<a class="admin__brand-link" href="/admin">MIEM Employees</a>' in template
assert ">Employees<" not in template
assert "/admin/employees" not in template
def test_directory_template_is_russian_and_uses_display_dates():
template = Path("app/templates/directory.html").read_text(encoding="utf-8")
assert "Сотрудники" in template
assert "Колонки" in template
assert "Применить" in template
assert "На странице: {{ value }}" in template
assert "{% for value in [25, 50, 100] %}" in template
assert "Найдено:" in template
assert "employee.first_seen_display" in template
assert "employee.last_seen_display" in template
assert "employee.dismissed_display" in template
assert "Directory" not in template
assert "employees found" not in template
def test_admin_employees_route_redirects_to_directory():
source = Path("app/admin.py").read_text(encoding="utf-8")
assert 'RedirectResponse("/admin/directory", status_code=303)' in source
def test_runs_template_links_to_run_detail():
template = Path("app/templates/runs.html").read_text(encoding="utf-8")
assert 'href="/admin/runs/{{ run.id }}"' in template
assert 'data-row-href="/admin/runs/{{ run.id }}"' in template
def test_run_detail_template_extends_base_and_shows_change_groups():
template = Path("app/templates/run_detail.html").read_text(encoding="utf-8")
assert '{% extends "base.html" %}' in template
assert 'id="new-employees"' in template
assert "Новые сотрудники" in template
assert "Потеряшки" in template
assert "Уволенные" in template
assert "Детализация сотрудников для этого запуска недоступна" in template
def test_dashboard_metric_cards_link_to_admin_targets():
template = Path("app/templates/dashboard.html").read_text(encoding="utf-8")
assert 'href="/admin/directory"' in template
assert 'href="/admin/directory?status=active"' in template
assert '/admin/runs/{{ latest_run.id }}#new-employees' in template
assert 'href="/admin/directory?status=dismissed"' in template
assert 'href="/admin/runs"' in template

View File

@@ -1,14 +1,19 @@
import time
from datetime import datetime, timezone from datetime import datetime, timezone
from types import SimpleNamespace
import jwt
from fastapi.testclient import TestClient from fastapi.testclient import TestClient
from cryptography.hazmat.primitives.asymmetric import rsa
from sqlalchemy import create_engine from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker from sqlalchemy.orm import sessionmaker
from sqlalchemy.pool import StaticPool from sqlalchemy.pool import StaticPool
import app.security as security
from app.config import Settings, get_settings from app.config import Settings, get_settings
from app.db import Base, get_db from app.db import Base, get_db
from app.main import app from app.main import app
from app.models import CrawlRun, Employee from app.models import CrawlRun, CrawlRunEmployeeChange, Employee
from app.security import SESSION_COOKIE, sign_session from app.security import SESSION_COOKIE, sign_session
@@ -18,7 +23,7 @@ def test_health_returns_versions():
response = client.get("/api/health") response = client.get("/api/health")
assert response.status_code == 200 assert response.status_code == 200
assert response.json()["backend_version"] == "0.2.4" assert response.json()["backend_version"] == "0.4.1"
def test_mcp_requires_token_and_lists_tools(): def test_mcp_requires_token_and_lists_tools():
@@ -38,7 +43,9 @@ def test_mcp_requires_token_and_lists_tools():
session.close() session.close()
app.dependency_overrides[get_db] = override_db app.dependency_overrides[get_db] = override_db
app.dependency_overrides[get_settings] = lambda: Settings(mcp_token="secret", session_secret="session-secret") app.dependency_overrides[get_settings] = lambda: Settings(
mcp_auth_mode="token", mcp_token="secret", session_secret="session-secret"
)
client = TestClient(app) client = TestClient(app)
unauthorized = client.post("/mcp", json={"jsonrpc": "2.0", "id": 1, "method": "tools/list", "params": {}}) unauthorized = client.post("/mcp", json={"jsonrpc": "2.0", "id": 1, "method": "tools/list", "params": {}})
@@ -51,6 +58,7 @@ def test_mcp_requires_token_and_lists_tools():
assert unauthorized.status_code == 401 assert unauthorized.status_code == 401
assert authorized.status_code == 200 assert authorized.status_code == 200
assert authorized.json()["result"]["tools"][0]["name"] == "search_employees" assert authorized.json()["result"]["tools"][0]["name"] == "search_employees"
assert any(tool["name"] == "get_crawl_run_details" for tool in authorized.json()["result"]["tools"])
app.dependency_overrides.clear() app.dependency_overrides.clear()
@@ -88,7 +96,9 @@ def test_mcp_search_employees_returns_matching_employee():
db.close() db.close()
app.dependency_overrides[get_db] = override_db app.dependency_overrides[get_db] = override_db
app.dependency_overrides[get_settings] = lambda: Settings(mcp_token="secret", session_secret="session-secret") app.dependency_overrides[get_settings] = lambda: Settings(
mcp_auth_mode="token", mcp_token="secret", session_secret="session-secret"
)
client = TestClient(app) client = TestClient(app)
response = client.post( response = client.post(
@@ -108,6 +118,218 @@ def test_mcp_search_employees_returns_matching_employee():
app.dependency_overrides.clear() app.dependency_overrides.clear()
def test_mcp_get_crawl_run_details_returns_changes():
engine = create_engine(
"sqlite:///:memory:",
connect_args={"check_same_thread": False},
poolclass=StaticPool,
)
Base.metadata.create_all(engine)
Session = sessionmaker(bind=engine)
session = Session()
run = CrawlRun(source_url="https://miem.hse.ru/persons", status="completed", new_count=1)
employee = Employee(
profile_key="staff:new",
profile_type="staff",
profile_id="new",
canonical_url="https://www.hse.ru/staff/new",
full_name="New Person",
status="active",
first_seen_at=datetime.now(timezone.utc),
last_seen_at=datetime.now(timezone.utc),
)
session.add_all([run, employee])
session.commit()
session.add(
CrawlRunEmployeeChange(
crawl_run_id=run.id,
employee_id=employee.id,
profile_key=employee.profile_key,
profile_url=employee.canonical_url,
full_name=employee.full_name,
change_type="new",
profile_available=True,
message="added",
)
)
session.commit()
run_id = run.id
session.close()
def override_db():
db = Session()
try:
yield db
finally:
db.close()
app.dependency_overrides[get_db] = override_db
app.dependency_overrides[get_settings] = lambda: Settings(
mcp_auth_mode="token", mcp_token="secret", session_secret="session-secret"
)
client = TestClient(app)
response = client.post(
"/mcp",
headers={"Authorization": "Bearer secret"},
json={
"jsonrpc": "2.0",
"id": 1,
"method": "tools/call",
"params": {"name": "get_crawl_run_details", "arguments": {"run_id": run_id}},
},
)
assert response.status_code == 200
text = response.json()["result"]["content"][0]["text"]
assert "New Person" in text
assert "changes_detail_available" in text
app.dependency_overrides.clear()
def test_mcp_oauth_rejects_static_token():
engine = create_engine(
"sqlite:///:memory:",
connect_args={"check_same_thread": False},
poolclass=StaticPool,
)
Base.metadata.create_all(engine)
Session = sessionmaker(bind=engine)
def override_db():
session = Session()
try:
yield session
finally:
session.close()
settings = Settings(
mcp_auth_mode="oauth",
mcp_token="secret",
session_secret="session-secret",
mcp_oauth_issuer="https://auth.example.com",
mcp_oauth_audience="miem-mcp",
mcp_oauth_jwks_url="https://auth.example.com/.well-known/jwks.json",
)
app.dependency_overrides[get_db] = override_db
app.dependency_overrides[get_settings] = lambda: settings
client = TestClient(app)
response = client.post(
"/mcp",
headers={"Authorization": "Bearer secret"},
json={"jsonrpc": "2.0", "id": 1, "method": "tools/list", "params": {}},
)
assert response.status_code == 401
assert response.headers["www-authenticate"] == (
'Bearer resource_metadata="http://localhost:8001/.well-known/oauth-protected-resource"'
)
app.dependency_overrides.clear()
def test_mcp_oauth_missing_auth_returns_metadata_challenge():
settings = Settings(
mcp_auth_mode="oauth",
mcp_resource_url="https://api.example.com/mcp",
mcp_oauth_issuer="https://auth.example.com",
mcp_oauth_audience="miem-mcp",
mcp_oauth_jwks_url="https://auth.example.com/.well-known/jwks.json",
)
app.dependency_overrides[get_settings] = lambda: settings
client = TestClient(app)
response = client.post("/mcp", json={"jsonrpc": "2.0", "id": 1, "method": "tools/list", "params": {}})
assert response.status_code == 401
assert response.headers["www-authenticate"] == (
'Bearer resource_metadata="https://api.example.com/.well-known/oauth-protected-resource"'
)
app.dependency_overrides.clear()
def test_mcp_accepts_valid_oauth_jwt(monkeypatch):
public_key, token = _oauth_key_and_token()
monkeypatch.setattr(security, "_get_mcp_oauth_signing_key", lambda _token, _settings: SimpleNamespace(key=public_key))
app.dependency_overrides[get_settings] = lambda: _oauth_settings()
client = TestClient(app)
response = client.post(
"/mcp",
headers={"Authorization": f"Bearer {token}"},
json={"jsonrpc": "2.0", "id": 1, "method": "tools/list", "params": {}},
)
assert response.status_code == 200
assert response.json()["result"]["tools"][0]["name"] == "search_employees"
app.dependency_overrides.clear()
def test_mcp_rejects_invalid_oauth_jwts(monkeypatch):
public_key, expired_token = _oauth_key_and_token(exp=int(time.time()) - 60)
_, wrong_issuer_token = _oauth_key_and_token(issuer="https://other.example.com")
_, wrong_audience_token = _oauth_key_and_token(audience="other-audience")
_, bad_signature_token = _oauth_key_and_token(public_key=public_key)
monkeypatch.setattr(security, "_get_mcp_oauth_signing_key", lambda _token, _settings: SimpleNamespace(key=public_key))
app.dependency_overrides[get_settings] = lambda: _oauth_settings()
client = TestClient(app)
for token in [expired_token, wrong_issuer_token, wrong_audience_token, bad_signature_token]:
response = client.post(
"/mcp",
headers={"Authorization": f"Bearer {token}"},
json={"jsonrpc": "2.0", "id": 1, "method": "tools/list", "params": {}},
)
assert response.status_code == 401
app.dependency_overrides.clear()
def test_mcp_rejects_oauth_jwt_without_required_scope(monkeypatch):
public_key, token = _oauth_key_and_token(scope="profile")
monkeypatch.setattr(security, "_get_mcp_oauth_signing_key", lambda _token, _settings: SimpleNamespace(key=public_key))
app.dependency_overrides[get_settings] = lambda: _oauth_settings()
client = TestClient(app)
response = client.post(
"/mcp",
headers={"Authorization": f"Bearer {token}"},
json={"jsonrpc": "2.0", "id": 1, "method": "tools/list", "params": {}},
)
assert response.status_code == 403
app.dependency_overrides.clear()
def test_mcp_protected_resource_metadata_uses_settings():
settings = Settings(
mcp_resource_url="https://api.example.com/mcp",
mcp_oauth_issuer="https://auth.example.com/",
mcp_oauth_required_scope="mcp:tools",
)
app.dependency_overrides[get_settings] = lambda: settings
client = TestClient(app)
response = client.get("/.well-known/oauth-protected-resource")
assert response.status_code == 200
assert response.json() == {
"resource": "https://api.example.com/mcp",
"authorization_servers": ["https://auth.example.com"],
"bearer_methods_supported": ["header"],
"scopes_supported": ["mcp:tools"],
"resource_documentation": "https://api.example.com/mcp",
}
app.dependency_overrides.clear()
def test_api_employees_and_stats_require_admin_session(): def test_api_employees_and_stats_require_admin_session():
engine = create_engine( engine = create_engine(
"sqlite:///:memory:", "sqlite:///:memory:",
@@ -130,8 +352,23 @@ def test_api_employees_and_stats_require_admin_session():
current_data={"contacts": {"emails": ["alpha@hse.ru"]}, "sections": []}, current_data={"contacts": {"emails": ["alpha@hse.ru"]}, "sections": []},
) )
) )
db.add(CrawlRun(source_url="https://miem.hse.ru/persons", status="completed", new_count=1)) run = CrawlRun(source_url="https://miem.hse.ru/persons", status="completed", new_count=1)
db.add(run)
db.commit() db.commit()
db.add(
CrawlRunEmployeeChange(
crawl_run_id=run.id,
employee_id=1,
profile_key="staff:alpha",
profile_url="https://www.hse.ru/staff/alpha",
full_name="Alpha Person",
change_type="new",
profile_available=True,
message="added",
)
)
db.commit()
run_id = run.id
db.close() db.close()
settings = Settings(admin_username="admin", admin_password="password", session_secret="session-secret") settings = Settings(admin_username="admin", admin_password="password", session_secret="session-secret")
@@ -150,10 +387,45 @@ def test_api_employees_and_stats_require_admin_session():
employees = client.get("/api/employees", params={"q": "Alpha", "has_email": True}) employees = client.get("/api/employees", params={"q": "Alpha", "has_email": True})
stats = client.get("/api/stats") stats = client.get("/api/stats")
run_details = client.get(f"/api/crawl-runs/{run_id}")
assert employees.status_code == 200 assert employees.status_code == 200
assert employees.json()["total"] == 1 assert employees.json()["total"] == 1
assert stats.status_code == 200 assert stats.status_code == 200
assert stats.json()["new_in_last_run"] == 1 assert stats.json()["new_in_last_run"] == 1
assert run_details.status_code == 200
assert run_details.json()["changes"]["new"][0]["full_name"] == "Alpha Person"
app.dependency_overrides.clear() app.dependency_overrides.clear()
def _oauth_settings() -> Settings:
return Settings(
mcp_auth_mode="oauth",
mcp_resource_url="https://api.example.com/mcp",
mcp_oauth_issuer="https://auth.example.com",
mcp_oauth_audience="miem-mcp",
mcp_oauth_jwks_url="https://auth.example.com/.well-known/jwks.json",
session_secret="session-secret",
)
def _oauth_key_and_token(
*,
issuer: str = "https://auth.example.com",
audience: str = "miem-mcp",
scope: str = "mcp:tools",
exp: int | None = None,
public_key=None,
):
private_key = rsa.generate_private_key(public_exponent=65537, key_size=2048)
claims = {
"iss": issuer,
"aud": audience,
"scope": scope,
"sub": "mcp-client",
"iat": int(time.time()),
"exp": exp or int(time.time()) + 300,
}
token = jwt.encode(claims, private_key, algorithm="RS256", headers={"kid": "test-key"})
return public_key or private_key.public_key(), token

21
tests/test_config.py Normal file
View File

@@ -0,0 +1,21 @@
import pytest
from pydantic import ValidationError
from app.config import Settings
def test_empty_crawl_limit_is_treated_as_none():
settings = Settings(crawl_limit="")
assert settings.crawl_limit is None
def test_numeric_crawl_limit_is_parsed():
settings = Settings(crawl_limit="25")
assert settings.crawl_limit == 25
def test_mcp_auth_mode_rejects_oauth_or_token_fallback():
with pytest.raises(ValidationError):
Settings(mcp_auth_mode="oauth_or_token")

View File

@@ -1,10 +1,25 @@
from datetime import datetime, timezone from datetime import datetime, timezone
from app.models import CrawlRun, Employee from app.models import CrawlRun, CrawlRunEmployeeChange, Employee
from app.services.crawler import _mark_dismissed, _upsert_employee from app.services.crawler import _mark_dismissed, _upsert_employee
def test_mark_dismissed_only_marks_missing_active_employees(db_session): class FakeResponse:
def __init__(self, status_code):
self.status_code = status_code
class FakeSession:
def __init__(self, statuses):
self.statuses = statuses
def get(self, url, **_kwargs):
return FakeResponse(self.statuses[url])
def test_mark_dismissed_records_missing_source_when_profile_is_available(db_session):
run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
db_session.add(run)
db_session.add( db_session.add(
Employee( Employee(
profile_key="staff:kept", profile_key="staff:kept",
@@ -16,8 +31,8 @@ def test_mark_dismissed_only_marks_missing_active_employees(db_session):
) )
db_session.add( db_session.add(
Employee( Employee(
profile_key="staff:gone", profile_key="staff:missing",
canonical_url="https://www.hse.ru/staff/gone", canonical_url="https://www.hse.ru/staff/missing",
status="active", status="active",
first_seen_at=datetime.now(timezone.utc), first_seen_at=datetime.now(timezone.utc),
last_seen_at=datetime.now(timezone.utc), last_seen_at=datetime.now(timezone.utc),
@@ -25,16 +40,53 @@ def test_mark_dismissed_only_marks_missing_active_employees(db_session):
) )
db_session.commit() db_session.commit()
dismissed = _mark_dismissed(db_session, {"staff:kept"}) dismissed = _mark_dismissed(
db_session,
run,
{"staff:kept"},
FakeSession({"https://www.hse.ru/staff/missing": 200}),
30,
)
assert dismissed == 0
assert db_session.query(Employee).filter_by(profile_key="staff:kept").one().status == "active"
missing = db_session.query(Employee).filter_by(profile_key="staff:missing").one()
assert missing.status == "active"
assert missing.dismissed_at is None
change = db_session.query(CrawlRunEmployeeChange).one()
assert change.change_type == "missing_from_source"
assert change.profile_available is True
def test_mark_dismissed_marks_missing_employee_when_profile_is_unavailable(db_session):
run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
employee = Employee(
profile_key="staff:gone",
canonical_url="https://www.hse.ru/staff/gone",
status="active",
first_seen_at=datetime.now(timezone.utc),
last_seen_at=datetime.now(timezone.utc),
)
db_session.add_all([run, employee])
db_session.commit()
dismissed = _mark_dismissed(
db_session,
run,
set(),
FakeSession({"https://www.hse.ru/staff/gone": 404}),
30,
)
assert dismissed == 1 assert dismissed == 1
assert db_session.query(Employee).filter_by(profile_key="staff:kept").one().status == "active" assert employee.status == "dismissed"
gone = db_session.query(Employee).filter_by(profile_key="staff:gone").one() assert employee.dismissed_at is not None
assert gone.status == "dismissed" change = db_session.query(CrawlRunEmployeeChange).one()
assert gone.dismissed_at is not None assert change.change_type == "dismissed"
assert change.profile_available is False
def test_upsert_employee_increments_new_count_for_new_employee(db_session): def test_upsert_employee_increments_new_count_and_records_change_for_new_employee(db_session):
run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running") run = CrawlRun(source_url="https://miem.hse.ru/persons", status="running")
db_session.add(run) db_session.add(run)
db_session.commit() db_session.commit()
@@ -56,3 +108,6 @@ def test_upsert_employee_increments_new_count_for_new_employee(db_session):
db_session.commit() db_session.commit()
assert run.new_count == 1 assert run.new_count == 1
change = db_session.query(CrawlRunEmployeeChange).one()
assert change.change_type == "new"
assert change.full_name == "New Person"

View File

@@ -9,7 +9,22 @@ def test_employee_detail_template_is_human_readable():
assert ">Tabs<" not in template assert ">Tabs<" not in template
assert "contacts.items" not in template assert "contacts.items" not in template
assert "contacts.contact_items" in template assert "contacts.contact_items" in template
assert "section.items" not in template
assert "section.list_items" in template
assert "Основная информация" in template assert "Основная информация" in template
assert "Контакты" in template assert "Контакты" in template
assert "Разделы профиля" in template assert "Разделы профиля" in template
assert "Snapshots" in template assert "graduation_theses" in template
assert "Год защиты" in template
assert "Parser version" not in template
assert "First seen" not in template
assert "Last seen" not in template
assert "Dismissed at" not in template
assert "Profile type" not in template
assert "Profile ID" not in template
assert "Впервые найден" in template
assert "Последний раз найден" in template
assert "Дата увольнения" in template
assert "Тип профиля" in template
assert "ID профиля" in template
assert "Снапшоты" in template

View File

@@ -1,9 +1,69 @@
from bs4 import BeautifulSoup from bs4 import BeautifulSoup
from app.parser.profile import extract_person_tabs from app.parser.profile import enrich_sections_from_hse_widgets, extract_person_tabs
from app.parser.profile_url import normalize_profile_url, parse_profile_identity from app.parser.profile_url import normalize_profile_url, parse_profile_identity
class FakeResponse:
def __init__(self, payload):
self.payload = payload
def raise_for_status(self):
return None
def json(self):
return self.payload
class FakeSession:
def __init__(self):
self.posts = []
self.gets = []
def post(self, url, **kwargs):
self.posts.append((url, kwargs))
return FakeResponse(
{
"status": "ok",
"result": {
"more": False,
"total": 1,
"items": [
{
"id": "888959076",
"type": "ARTICLE",
"title": "Дублирование пакетов",
"year": 2023,
"description": {"short": {"ru": "Информационные процессы. 2023."}},
}
],
},
}
)
def get(self, url, **kwargs):
self.gets.append((url, kwargs))
return FakeResponse(
{
"lang": "ru",
"success": True,
"data": [
{
"id": 1045750164,
"year": 2025,
"level": "Бакалавриат",
"title": "Аппаратно-программный комплекс защиты сети",
"rating": 8,
"student": "Лесняк Владислав Евгеньевич",
"learnProgram": {"title": "Информатика и вычислительная техника", "url": "https://hse.ru/ba/isct/"},
"orgUnit": {"title": "МИЭМ", "url": "https://www.hse.ru/org/url/59315150"},
"supervisors": [{"url": "https://www.hse.ru/org/persons/803294906", "name": "Борисов Сергей Петрович"}],
}
],
}
)
def test_normalize_profile_url_supports_staff_and_org_persons(): def test_normalize_profile_url_supports_staff_and_org_persons():
assert normalize_profile_url("/staff/avsergeev#sci") == "https://www.hse.ru/staff/avsergeev" assert normalize_profile_url("/staff/avsergeev#sci") == "https://www.hse.ru/staff/avsergeev"
assert normalize_profile_url("https://www.hse.ru/org/persons/123/") == "https://www.hse.ru/org/persons/123" assert normalize_profile_url("https://www.hse.ru/org/persons/123/") == "https://www.hse.ru/org/persons/123"
@@ -26,3 +86,34 @@ def test_extract_person_tabs_prefers_person_menu_addition():
assert [tab["title"] for tab in tabs] == ["Домашняя страница", "Публикации"] assert [tab["title"] for tab in tabs] == ["Домашняя страница", "Публикации"]
assert tabs[1]["href"] == "https://www.hse.ru/staff/avsergeev#sci" assert tabs[1]["href"] == "https://www.hse.ru/staff/avsergeev#sci"
def test_enrich_sections_from_hse_widgets_loads_publications_and_vkr():
soup = BeautifulSoup(
"""
<script src="/n/stat/publications/dist-w/publs.js" data-author="568398853" data-widget-name="AuthorSearch"></script>
<script src="/n/stat/vkr/app.js" data-api-url="/n/vkr/api/" data-person-id="803294906"></script>
""",
"html.parser",
)
session = FakeSession()
sections = enrich_sections_from_hse_widgets(
session,
soup,
"https://www.hse.ru/org/persons/803294906",
{"User-Agent": "test"},
10,
[],
)
publications = next(section for section in sections if section["type"] == "publications")
theses = next(section for section in sections if section["type"] == "graduation_theses")
assert publications["publications_count"] == 1
assert publications["publications"][0]["url"] == "https://publications.hse.ru/view/888959076"
assert theses["theses_count"] == 1
assert theses["theses"][0]["student"] == "Лесняк Владислав Евгеньевич"
assert theses["theses"][0]["project_url"] == "https://www.hse.ru/edu/vkr/1045750164"
assert session.posts[0][0] == "https://publications.hse.ru/api/searchPubs"
assert session.gets[0][1]["params"] == {"supervisorId": "803294906"}