docs: add scheduled worker changelog

Reviewed-on: #33

.env.example

@@ -15,4 +15,3 @@ ADMIN_USERNAME=admin
 ADMIN_PASSWORD=change-me
 SESSION_SECRET=change-me-session-secret
 API_PORT=8000
-MCP_PORT=8001

CHANGELOG.md

@@ -1,5 +1,25 @@
 # Changelog
 
+## 0.7.6
+
+- Возвращён еженедельный автоматический запуск обхода сотрудников.
+
+## 0.7.5
+
+- Production Compose запускает только API и PostgreSQL.
+
+## 0.7.4
+
+- Ускорена фильтрация сотрудников по учёной степени.
+
+## 0.7.3
+
+- Восстановлена проверка профилей сотрудников и хранение истории URL профиля.
+
+## 0.7.2
+
+- В каталоге сотрудников добавлены фильтр и колонка учёной степени.
+
 ## 0.7.1
 
 - Добавлена кнопка «Проверить уволенных» для принудительной сверки статуса уволенных сотрудников с текущим списком источника.

MCP_DESCRIPTION.md

@@ -8,7 +8,7 @@
 - Подключение к приложению: `app/main.py`
 - HTTP endpoint: `POST /mcp`
 - Локально при обычном запуске API: `http://localhost:8000/mcp`
-- В Docker Compose через отдельный сервис `mcp`: `http://localhost:8001/mcp`
+- В Docker Compose endpoint обслуживает `api`: `http://localhost:8000/mcp`
 - Авторизация на уровне приложения: отсутствует. Заголовок `Authorization` не проверяется и не влияет на ответ.
 
 Если доступ к MCP нужно ограничить, это должно делаться внешним контуром: bind на localhost, VPN, firewall, reverse proxy или отдельная сетевая политика.
@@ -622,7 +622,7 @@ Hash набора считается по отсортированному сп
 Список tools:
 
 ```bash
-curl http://localhost:8001/mcp \
+curl http://localhost:8000/mcp \
   -H "Content-Type: application/json" \
   -d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}'
 ```
@@ -630,7 +630,7 @@ curl http://localhost:8001/mcp \
 Поиск сотрудника:
 
 ```bash
-curl http://localhost:8001/mcp \
+curl http://localhost:8000/mcp \
   -H "Content-Type: application/json" \
   -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"search_employees","arguments":{"query":"Сергеев","limit":5}}}'
 ```
@@ -638,7 +638,7 @@ curl http://localhost:8001/mcp \
 Полная синхронизация:
 
 ```bash
-curl http://localhost:8001/mcp \
+curl http://localhost:8000/mcp \
   -H "Content-Type: application/json" \
   -d '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"sync_employees","arguments":{"include_data":false}}}'
 ```
@@ -646,7 +646,7 @@ curl http://localhost:8001/mcp \
 Delta-синхронизация:
 
 ```bash
-curl http://localhost:8001/mcp \
+curl http://localhost:8000/mcp \
   -H "Content-Type: application/json" \
   -d '{"jsonrpc":"2.0","id":4,"method":"tools/call","params":{"name":"sync_employees","arguments":{"client_hash":"known-sha256","include_data":true}}}'
 ```

README.md

@@ -4,9 +4,8 @@
 
 ## Архитектура
 
-- `api`: FastAPI, REST API, HTML-админка, healthcheck.
-- `worker`: weekly scheduler, который запускает парсинг по `CRAWL_CRON`.
-- `mcp`: открытый HTTP MCP endpoint для ИИ-агентов.
+- `api`: FastAPI, REST API, HTML-админка, MCP endpoint и healthcheck.
+- `worker`: weekly scheduler, который запускает парсинг по `CRAWL_CRON`.
 - `postgres`: основная БД.
 
 Парсер использует фиксированный источник сотрудников, по умолчанию `https://miem.hse.ru/persons`. Для каждой карточки сохраняются ФИО, должности, год начала работы, контакты, идентификаторы, вкладки профиля, секции, публикации, курсы, ВКР, новости, JSON-снапшот и сжатый HTML-снапшот. Детальные публикации дополнительно нормализуются в отдельную таблицу `employee_publications`, а новости из блока «В новостях» — в `employee_news_links`. Ссылки обходятся только из меню профиля самого сотрудника (`person-menu`), например `#sci`, `#teaching`, `#main`.
@@ -22,8 +21,8 @@ cp .env.example .env
 Основные настройки:
 
 - `DATABASE_URL`: строка подключения SQLAlchemy.
-- `SOURCE_URL`: список сотрудников МИЭМ.
-- `CRAWL_CRON`: расписание в формате crontab, по умолчанию `0 3 * * 1`.
+- `SOURCE_URL`: список сотрудников МИЭМ.
+- `CRAWL_CRON`: расписание в формате crontab, по умолчанию `0 3 * * 1`.
 - `CRAWL_LIMIT`: опциональный лимит профилей для тестового запуска.
 - `ADMIN_USERNAME`, `ADMIN_PASSWORD`: логин и пароль админки.
 - `SESSION_SECRET`: секрет подписи cookie.
@@ -51,13 +50,12 @@ uvicorn app.main:app --reload
 ## Docker Compose
 
 ```bash
-docker compose up --build
+docker compose up -d --build --remove-orphans
 ```
 
 По умолчанию:
 
 - API и админка: `http://localhost:8000`
-- MCP: `http://localhost:8001/mcp`
 - Postgres: `localhost:5432`
 
 Таблицы создаются приложением при старте. При обновлении существующей базы приложение также добавляет недостающие runtime-колонки, например `crawl_runs.skipped_count`. SQL-миграции для ручного применения лежат в `migrations/`.
@@ -84,7 +82,7 @@ docker compose up --build
 
 ## Парсинг
 
-Weekly worker запускается по `CRAWL_CRON`. Ручной запуск доступен в админке на `Dashboard` и странице `Runs` или через REST:
+Worker запускает обход по `CRAWL_CRON`. Ручной запуск также доступен в админке на `Dashboard` и странице `Runs` или через REST:
 
 ```bash
 curl -X POST http://localhost:8000/api/crawl-runs --cookie "miem_admin_session=..."
@@ -131,7 +129,7 @@ Endpoint: `POST /mcp`, без авторизации на уровне прил
 Пример локального запроса списка tools:
 
 ```bash
-curl http://localhost:8001/mcp \
+curl http://localhost:8000/mcp \
   -H "Content-Type: application/json" \
   -d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}'
 ```
@@ -141,10 +139,10 @@ curl http://localhost:8001/mcp \
 ## Обслуживание
 
 ```bash
-docker compose logs -f api
-docker compose logs -f worker
+docker compose logs -f api
+docker compose logs -f worker
 docker compose exec postgres pg_dump -U miem miem_workers > backup.sql
 docker compose down
 ```
 
-Версия сервиса: `0.7.1`. Админка всегда показывает версии backend и frontend в footer.
+Версия сервиса: `0.7.6`. Админка всегда показывает версии backend и frontend в footer.

app/admin.py

@@ -78,8 +78,9 @@ def directory(
     status: str | None = None,
     q: str | None = None,
     started_from: str | None = None,
-    started_to: str | None = None,
-    has_email: str | None = None,
+    started_to: str | None = None,
+    has_email: str | None = None,
+    has_academic_degree: str | None = None,
     sort: str = "full_name",
     direction: str = "asc",
     limit: int = 50,
@@ -90,14 +91,16 @@ def directory(
     require_admin(request, settings)
     parsed_started_from = _parse_date(started_from)
     parsed_started_to = _parse_date(started_to)
-    parsed_has_email = None if has_email in (None, "") else has_email == "true"
+    parsed_has_email = None if has_email in (None, "") else has_email == "true"
+    parsed_has_academic_degree = None if has_academic_degree in (None, "") else has_academic_degree == "true"
     page = list_employees_page(
         db,
         status=status,
         q=q,
         started_from=parsed_started_from,
         started_to=parsed_started_to,
-        has_email=parsed_has_email,
+        has_email=parsed_has_email,
+        has_academic_degree=parsed_has_academic_degree,
         sort=sort,
         direction=direction,
         limit=limit,
@@ -113,7 +116,8 @@ def directory(
                 "q": q or "",
                 "started_from": started_from or "",
                 "started_to": started_to or "",
-                "has_email": has_email or "",
+                "has_email": has_email or "",
+                "has_academic_degree": has_academic_degree or "",
                 "sort": sort,
                 "direction": direction,
                 "limit": page["limit"],

app/api.py

@@ -28,6 +28,7 @@ def list_employees(
     started_from: date | None = None,
     started_to: date | None = None,
     has_email: bool | None = None,
+    has_academic_degree: bool | None = None,
     sort: str = "full_name",
     direction: str = "asc",
     limit: int = 50,
@@ -43,6 +44,7 @@ def list_employees(
         started_from=started_from,
         started_to=started_to,
         has_email=has_email,
+        has_academic_degree=has_academic_degree,
         sort=sort,
         direction=direction,
         limit=limit,

app/db.py

@@ -41,6 +41,30 @@ def _ensure_runtime_schema() -> None:
         models.EmployeeNewsLink.__table__.create(bind=engine, checkfirst=True)
         inspector = inspect(engine)
         table_names = set(inspector.get_table_names())
+    if "employees" in table_names and "employee_profile_urls" not in table_names:
+        models.EmployeeProfileUrl.__table__.create(bind=engine, checkfirst=True)
+        inspector = inspect(engine)
+    if "employees" in table_names:
+        employee_columns = {column["name"] for column in inspector.get_columns("employees")}
+        missing_columns = []
+        if "profile_unavailable_streak" not in employee_columns:
+            missing_columns.append("profile_unavailable_streak INTEGER NOT NULL DEFAULT 0")
+        if "last_profile_check_at" not in employee_columns:
+            missing_columns.append("last_profile_check_at TIMESTAMPTZ")
+        if "has_academic_degree" not in employee_columns:
+            missing_columns.append("has_academic_degree BOOLEAN NOT NULL DEFAULT FALSE")
+        if missing_columns:
+            with engine.begin() as connection:
+                for column in missing_columns:
+                    connection.execute(text(f"ALTER TABLE employees ADD COLUMN {column}"))
+                if engine.dialect.name == "postgresql":
+                    connection.execute(
+                        text(
+                            "UPDATE employees SET has_academic_degree = "
+                            "COALESCE(current_data::text ~* '(кандидат|доктор).{0,80}наук|ph\\.?[[:space:]]*d\\.?', FALSE)"
+                        )
+                    )
+                connection.execute(text("CREATE INDEX IF NOT EXISTS ix_employees_has_academic_degree ON employees (has_academic_degree)"))
     if "crawl_runs" not in table_names:
         return
     crawl_run_columns = {column["name"] for column in inspector.get_columns("crawl_runs")}

app/models.py

@@ -1,6 +1,6 @@
 from datetime import datetime, timezone
 
-from sqlalchemy import DateTime, ForeignKey, Index, Integer, LargeBinary, String, Text, UniqueConstraint
+from sqlalchemy import Boolean, DateTime, ForeignKey, Index, Integer, LargeBinary, String, Text, UniqueConstraint
 from sqlalchemy.dialects.postgresql import JSONB
 from sqlalchemy.orm import Mapped, mapped_column, relationship
 from sqlalchemy.types import JSON
@@ -21,6 +21,7 @@ class Employee(Base):
         UniqueConstraint("profile_key", name="uq_employees_profile_key"),
         Index("ix_employees_full_name", "full_name"),
         Index("ix_employees_status", "status"),
+        Index("ix_employees_has_academic_degree", "has_academic_degree"),
     )
 
     id: Mapped[int] = mapped_column(Integer, primary_key=True)
@@ -33,19 +34,39 @@ class Employee(Base):
     first_seen_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow, nullable=False)
     last_seen_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow, nullable=False)
     dismissed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
+    profile_unavailable_streak: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
+    last_profile_check_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
     parser_version: Mapped[str | None] = mapped_column(String(32))
     current_data: Mapped[dict | None] = mapped_column(json_type)
+    has_academic_degree: Mapped[bool] = mapped_column(Boolean, default=False, nullable=False)
     current_checksum: Mapped[str | None] = mapped_column(String(64))
     created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow, nullable=False)
     updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow, onupdate=utcnow, nullable=False)
 
     snapshots: Mapped[list["EmployeeSnapshot"]] = relationship(back_populates="employee")
+    profile_urls: Mapped[list["EmployeeProfileUrl"]] = relationship(back_populates="employee", cascade="all, delete-orphan")
     tabs: Mapped[list["ProfileTab"]] = relationship(back_populates="employee", cascade="all, delete-orphan")
     publications: Mapped[list["EmployeePublication"]] = relationship(back_populates="employee", cascade="all, delete-orphan")
     news_links: Mapped[list["EmployeeNewsLink"]] = relationship(back_populates="employee", cascade="all, delete-orphan")
     crawl_run_changes: Mapped[list["CrawlRunEmployeeChange"]] = relationship(back_populates="employee")
 
 
+class EmployeeProfileUrl(Base):
+    __tablename__ = "employee_profile_urls"
+    __table_args__ = (
+        UniqueConstraint("employee_id", "url", name="uq_employee_profile_urls_employee_url"),
+        Index("ix_employee_profile_urls_employee_id", "employee_id"),
+    )
+
+    id: Mapped[int] = mapped_column(Integer, primary_key=True)
+    employee_id: Mapped[int] = mapped_column(ForeignKey("employees.id", ondelete="CASCADE"), nullable=False)
+    url: Mapped[str] = mapped_column(Text, nullable=False)
+    first_seen_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow, nullable=False)
+    last_seen_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow, nullable=False)
+
+    employee: Mapped[Employee] = relationship(back_populates="profile_urls")
+
+
 class EmployeeSnapshot(Base):
     __tablename__ = "employee_snapshots"
     __table_args__ = (Index("ix_employee_snapshots_employee_id", "employee_id"),)

app/services/academic_degrees.py

@@ -0,0 +1,23 @@
+import re
+from typing import Any
+
+
+_PATTERN = re.compile(r"\b(?:кандидат|доктор)\s+[\w\s-]{0,80}?\s+наук\b|\bph\.?\s*d\.?\b", re.IGNORECASE)
+
+
+def academic_degrees(data: dict[str, Any] | None) -> list[str]:
+    degrees = []
+    for section in (data or {}).get("sections") or []:
+        if not isinstance(section, dict) or not re.search(r"уч[её]н.*степен|academic degree", str(section.get("title") or ""), re.IGNORECASE):
+            continue
+        values = [
+            *(entry.get("text") for entry in section.get("year_entries") or [] if isinstance(entry, dict)),
+            *(section.get("paragraphs") or []),
+            *(section.get("items") or []),
+            section.get("raw_text"),
+        ]
+        for value in values:
+            text = str(value or "").strip()
+            if text and _PATTERN.search(text) and text not in degrees:
+                degrees.append(text)
+    return degrees

app/services/admin_data.py

@@ -9,6 +9,7 @@ from sqlalchemy import Select, Text, and_, desc, func, or_, select
 from sqlalchemy.orm import Session
 
 from app.models import CrawlError, CrawlRun, CrawlRunEmployeeChange, Employee, EmployeeNewsLink
+from app.services.academic_degrees import academic_degrees
 
 EMPLOYEE_SORTS = {
     "full_name": Employee.full_name,
@@ -19,7 +20,6 @@ EMPLOYEE_SORTS = {
     "hse_start_year": Employee.current_data["hse_start_year"].as_integer(),
 }
 
-
 def employee_display_payload(employee: Employee) -> dict[str, Any]:
     data = _as_dict(employee.current_data)
     contacts = _as_dict(data.get("contacts"))
@@ -28,6 +28,7 @@ def employee_display_payload(employee: Employee) -> dict[str, Any]:
     positions = _clean_list(data.get("positions"))
     emails = _clean_list(contacts.get("emails"))
     phones = _clean_list(contacts.get("phones"))
+    degree_values = academic_degrees(data)
     return {
         "id": employee.id,
         "full_name": employee.full_name,
@@ -42,6 +43,7 @@ def employee_display_payload(employee: Employee) -> dict[str, Any]:
         "phones": phones,
         "phone_text": ", ".join(phones),
         "address": contacts.get("address"),
+        "academic_degree_text": "; ".join(degree_values),
         "publications_count": _count_section_items(sections, "publications"),
         "courses_count": _count_section_items(sections, "courses_by_year"),
         "news_count": len(stored_news_links) or _count_section_items(sections, "news"),
@@ -81,6 +83,7 @@ def build_employee_query(
     started_from: date | None = None,
     started_to: date | None = None,
     has_email: bool | None = None,
+    has_academic_degree: bool | None = None,
 ) -> Select[tuple[Employee]]:
     stmt = select(Employee)
     filters = []
@@ -97,6 +100,8 @@ def build_employee_query(
         filters.append(Employee.current_data.cast(Text).ilike("%@%"))
     elif has_email is False:
         filters.append(or_(Employee.current_data.is_(None), ~Employee.current_data.cast(Text).ilike("%@%")))
+    if has_academic_degree is not None:
+        filters.append(Employee.has_academic_degree.is_(has_academic_degree))
     if filters:
         stmt = stmt.where(and_(*filters))
     return stmt
@@ -110,6 +115,7 @@ def list_employees_page(
     started_from: date | None = None,
     started_to: date | None = None,
     has_email: bool | None = None,
+    has_academic_degree: bool | None = None,
     sort: str = "full_name",
     direction: str = "asc",
     limit: int = 50,
@@ -123,6 +129,7 @@ def list_employees_page(
         started_from=started_from,
         started_to=started_to,
         has_email=has_email,
+        has_academic_degree=has_academic_degree,
     )
     total = db.scalar(select(func.count()).select_from(base_stmt.subquery())) or 0
     sort_column = EMPLOYEE_SORTS.get(sort, Employee.full_name)
@@ -145,6 +152,10 @@ def stats_payload(db: Session) -> dict[str, Any]:
     return {
         "total": db.scalar(select(func.count()).select_from(Employee)) or 0,
         "active": db.scalar(select(func.count()).select_from(Employee).where(Employee.status == "active")) or 0,
+        "verification_required": db.scalar(
+            select(func.count()).select_from(Employee).where(Employee.status == "verification_required")
+        )
+        or 0,
         "dismissed": db.scalar(select(func.count()).select_from(Employee).where(Employee.status == "dismissed")) or 0,
         "new_in_last_run": latest_run.new_count if latest_run else 0,
         "latest_added": employee_display_payload(latest_added) if latest_added else None,
@@ -188,7 +199,7 @@ def run_detail_payload(db: Session, run: CrawlRun | None) -> dict[str, Any] | No
         .order_by(CrawlRunEmployeeChange.created_at, CrawlRunEmployeeChange.id)
     ).all()
     errors = db.scalars(select(CrawlError).where(CrawlError.crawl_run_id == run.id).order_by(CrawlError.created_at)).all()
-    grouped_changes = {"new": [], "missing_from_source": [], "dismissed": []}
+    grouped_changes = {"new": [], "missing_from_source": [], "verification_required": [], "dismissed": []}
     for change in changes:
         grouped_changes.setdefault(change.change_type, []).append(_change_payload(change))
     return {
@@ -215,7 +226,7 @@ def format_admin_datetime(value: Any) -> str:
 
 
 def _employee_status_display(status: str | None) -> str:
-    labels = {"active": "Работает", "dismissed": "Уволен"}
+    labels = {"active": "Работает", "verification_required": "Требует проверки", "dismissed": "Уволен"}
     return labels.get(status or "", status or "Не указано")
 
 

app/services/crawler.py

@@ -24,7 +24,8 @@ from app.models import (
 )
 from app.parser.collector import collect_profile_links
 from app.parser.profile import parse_person_profile
-from app.parser.profile_url import profile_key
+from app.parser.profile_url import profile_key
+from app.services.academic_degrees import academic_degrees
 from app.services.dataset_versions import get_or_create_current_version
 from app.services.resource_cache import ResourceCache
 
@@ -264,8 +265,9 @@ def _upsert_employee(db: Session, run: CrawlRun, parsed: dict) -> tuple[Employee
     employee.dismissed_at = None
     employee.profile_unavailable_streak = 0
     employee.last_profile_check_at = now
-    employee.parser_version = parser_version
-    if changed:
+    employee.parser_version = parser_version
+    employee.has_academic_degree = bool(academic_degrees(parsed))
+    if changed:
         employee.current_data = parsed
     employee.current_checksum = checksum
     db.flush()

app/static/admin.js

@@ -5,6 +5,7 @@
     "positions",
     "hse_start_year",
     "email",
+    "academic_degree",
     "last_seen_at",
     "dismissed_at",
     "profile",

app/templates/directory.html

@@ -15,6 +15,7 @@
     <select class="directory__input" name="status">
       <option value="" {% if not filters.status %}selected{% endif %}>Все статусы</option>
       <option value="active" {% if filters.status == "active" %}selected{% endif %}>Работает</option>
+      <option value="verification_required" {% if filters.status == "verification_required" %}selected{% endif %}>Требует проверки</option>
       <option value="dismissed" {% if filters.status == "dismissed" %}selected{% endif %}>Уволен</option>
     </select>
     <select class="directory__input" name="has_email">
@@ -22,6 +23,11 @@
       <option value="true" {% if filters.has_email == "true" %}selected{% endif %}>Есть email</option>
       <option value="false" {% if filters.has_email == "false" %}selected{% endif %}>Нет email</option>
     </select>
+    <select class="directory__input" name="has_academic_degree" aria-label="Учёная степень">
+      <option value="" {% if not filters.has_academic_degree %}selected{% endif %}>Любая учёная степень</option>
+      <option value="true" {% if filters.has_academic_degree == "true" %}selected{% endif %}>Есть учёная степень</option>
+      <option value="false" {% if filters.has_academic_degree == "false" %}selected{% endif %}>Нет учёной степени</option>
+    </select>
     <input class="directory__input" type="date" name="started_from" value="{{ filters.started_from }}" aria-label="Впервые найден с">
     <input class="directory__input" type="date" name="started_to" value="{{ filters.started_to }}" aria-label="Впервые найден по">
     <select class="directory__input" name="sort">
@@ -53,6 +59,7 @@
           <th class="directory-table__head" data-column="email">Email</th>
           <th class="directory-table__head" data-column="phone">Телефон</th>
           <th class="directory-table__head" data-column="address">Адрес</th>
+          <th class="directory-table__head" data-column="academic_degree">Учёная степень</th>
           <th class="directory-table__head" data-column="publications_count">Публикации</th>
           <th class="directory-table__head" data-column="courses_count">Курсы</th>
           <th class="directory-table__head" data-column="news_count">Новости</th>
@@ -72,6 +79,7 @@
           <td class="directory-table__cell" data-column="email">{{ employee.email_text }}</td>
           <td class="directory-table__cell" data-column="phone">{{ employee.phone_text }}</td>
           <td class="directory-table__cell" data-column="address">{{ employee.address or "" }}</td>
+          <td class="directory-table__cell" data-column="academic_degree">{{ employee.academic_degree_text }}</td>
           <td class="directory-table__cell" data-column="publications_count">{{ employee.publications_count }}</td>
           <td class="directory-table__cell" data-column="courses_count">{{ employee.courses_count }}</td>
           <td class="directory-table__cell" data-column="news_count">{{ employee.news_count }}</td>
@@ -81,7 +89,7 @@
           <td class="directory-table__cell" data-column="profile"><a class="admin__link" href="{{ employee.canonical_url }}">Открыть</a></td>
         </tr>
         {% else %}
-        <tr><td class="directory-table__empty" colspan="14">По этим фильтрам сотрудники не найдены.</td></tr>
+        <tr><td class="directory-table__empty" colspan="15">По этим фильтрам сотрудники не найдены.</td></tr>
         {% endfor %}
       </tbody>
     </table>
@@ -108,7 +116,7 @@
       <button class="button button--ghost" type="button" data-columns-close>Закрыть</button>
     </div>
     <div class="columns-modal__grid">
-      {% for key, label in [("full_name", "ФИО"), ("status", "Статус"), ("positions", "Должности"), ("hse_start_year", "Год начала"), ("email", "Email"), ("phone", "Телефон"), ("address", "Адрес"), ("publications_count", "Публикации"), ("courses_count", "Курсы"), ("news_count", "Новости"), ("first_seen_at", "Впервые найден"), ("last_seen_at", "Последний раз найден"), ("dismissed_at", "Дата увольнения"), ("profile", "Профиль")] %}
+      {% for key, label in [("full_name", "ФИО"), ("status", "Статус"), ("positions", "Должности"), ("hse_start_year", "Год начала"), ("email", "Email"), ("phone", "Телефон"), ("address", "Адрес"), ("academic_degree", "Учёная степень"), ("publications_count", "Публикации"), ("courses_count", "Курсы"), ("news_count", "Новости"), ("first_seen_at", "Впервые найден"), ("last_seen_at", "Последний раз найден"), ("dismissed_at", "Дата увольнения"), ("profile", "Профиль")] %}
       <label class="columns-modal__option"><input class="columns-modal__checkbox" type="checkbox" value="{{ key }}" data-column-toggle> {{ label }}</label>
       {% endfor %}
     </div>

app/templates/run_detail.html

@@ -23,7 +23,7 @@
   {% endif %}
 </section>
 
-{% for group, title in [("new", "Новые сотрудники"), ("missing_from_source", "Потеряшки"), ("dismissed", "Уволенные")] %}
+{% for group, title in [("new", "Новые сотрудники"), ("missing_from_source", "Потеряшки"), ("verification_required", "Требуют проверки"), ("dismissed", "Уволенные")] %}
 <section class="panel"{% if group == "new" %} id="new-employees"{% endif %}>
   <h2 class="panel__title">{{ title }}</h2>
   {% set items = run.changes[group] %}

app/version.py

@@ -1,3 +1,3 @@
-APP_VERSION = "0.7.1"
-FRONTEND_VERSION = "0.7.1"
-BACKEND_VERSION = "0.7.1"
+APP_VERSION = "0.7.6"
+FRONTEND_VERSION = "0.7.6"
+BACKEND_VERSION = "0.7.6"

docker-compose.yml

@@ -35,17 +35,5 @@ services:
       postgres:
         condition: service_healthy
 
-  mcp:
-    build: .
-    command: uvicorn app.main:app --host 0.0.0.0 --port 8000
-    env_file: .env
-    environment:
-      DATABASE_URL: postgresql+psycopg://${POSTGRES_USER:-miem}:${POSTGRES_PASSWORD:-miem_password}@postgres:5432/${POSTGRES_DB:-miem_workers}
-    ports:
-      - "127.0.0.1:${MCP_PORT:-8001}:8000"
-    depends_on:
-      postgres:
-        condition: service_healthy
-
 volumes:
   postgres_data:

migrations/001_init.sql

@@ -33,6 +33,7 @@ CREATE TABLE IF NOT EXISTS employees (
   dismissed_at TIMESTAMPTZ,
   parser_version VARCHAR(32),
   current_data JSONB,
+  has_academic_degree BOOLEAN NOT NULL DEFAULT FALSE,
   current_checksum VARCHAR(64),
   created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
   updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
@@ -40,6 +41,7 @@ CREATE TABLE IF NOT EXISTS employees (
 
 CREATE INDEX IF NOT EXISTS ix_employees_full_name ON employees (full_name);
 CREATE INDEX IF NOT EXISTS ix_employees_status ON employees (status);
+CREATE INDEX IF NOT EXISTS ix_employees_has_academic_degree ON employees (has_academic_degree);
 
 CREATE TABLE IF NOT EXISTS employee_snapshots (
   id SERIAL PRIMARY KEY,

migrations/008_profile_verification.sql

@@ -0,0 +1,17 @@
+ALTER TABLE employees
+ADD COLUMN IF NOT EXISTS profile_unavailable_streak INTEGER NOT NULL DEFAULT 0;
+
+ALTER TABLE employees
+ADD COLUMN IF NOT EXISTS last_profile_check_at TIMESTAMPTZ;
+
+CREATE TABLE IF NOT EXISTS employee_profile_urls (
+  id SERIAL PRIMARY KEY,
+  employee_id INTEGER NOT NULL REFERENCES employees(id) ON DELETE CASCADE,
+  url TEXT NOT NULL,
+  first_seen_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+  last_seen_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+  CONSTRAINT uq_employee_profile_urls_employee_url UNIQUE (employee_id, url)
+);
+
+CREATE INDEX IF NOT EXISTS ix_employee_profile_urls_employee_id
+  ON employee_profile_urls (employee_id);

migrations/009_academic_degree_filter.sql

@@ -0,0 +1,11 @@
+ALTER TABLE employees
+ADD COLUMN IF NOT EXISTS has_academic_degree BOOLEAN NOT NULL DEFAULT FALSE;
+
+UPDATE employees
+SET has_academic_degree = COALESCE(
+  current_data::text ~* '(кандидат|доктор).{0,80}наук|ph\.?[[:space:]]*d\.?',
+  FALSE
+);
+
+CREATE INDEX IF NOT EXISTS ix_employees_has_academic_degree
+  ON employees (has_academic_degree);

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "miem-workers"
-version = "0.7.1"
+version = "0.7.6"
 description = "MIEM employees parser, admin API, and MCP server"
 requires-python = ">=3.11"
 dependencies = [

tests/test_admin_data.py

@@ -51,6 +51,51 @@ def test_employee_display_payload_extracts_common_fields(db_session):
     assert payload["first_seen_display"] != "Не указано"
 
 
+def test_list_employees_page_filters_and_displays_academic_degrees(db_session):
+    db_session.add_all(
+        [
+            Employee(
+                profile_key="staff:degree",
+                canonical_url="https://www.hse.ru/staff/degree",
+                full_name="Doctor",
+                status="active",
+                first_seen_at=datetime.now(timezone.utc),
+                last_seen_at=datetime.now(timezone.utc),
+                has_academic_degree=True,
+                current_data={
+                    "sections": [
+                        {
+                            "title": "Образование и учёные степени",
+                            "year_entries": [{"year": 2020, "text": "Доктор технических наук"}],
+                        }
+                    ]
+                },
+            ),
+            Employee(
+                profile_key="staff:no-degree",
+                canonical_url="https://www.hse.ru/staff/no-degree",
+                full_name="Master",
+                status="active",
+                first_seen_at=datetime.now(timezone.utc),
+                last_seen_at=datetime.now(timezone.utc),
+                has_academic_degree=False,
+                current_data={"sections": [{"title": "Образование", "items": ["Магистратура"]}]},
+            ),
+        ]
+    )
+    db_session.commit()
+
+    page = list_employees_page(db_session, has_academic_degree=True)
+
+    assert page["total"] == 1
+    assert page["employees"][0]["full_name"] == "Doctor"
+    assert page["employees"][0]["academic_degree_text"] == "Доктор технических наук"
+
+    page = list_employees_page(db_session, has_academic_degree=False)
+
+    assert page["employees"][0]["full_name"] == "Master"
+
+
 def test_employee_detail_payload_normalizes_human_readable_sections(db_session):
     employee = Employee(
         profile_key="staff:person",

tests/test_admin_templates.py

@@ -22,7 +22,9 @@ def test_directory_template_is_russian_and_uses_display_dates():
     assert "На странице: {{ value }}" in template
     assert "{% for value in [25, 50, 100] %}" in template
     assert "Найдено:" in template
-    assert "Новости" in template
+    assert "Новости" in template
+    assert "Есть учёная степень" in template
+    assert 'data-column="academic_degree"' in template
     assert "employee.news_count" in template
     assert "employee.first_seen_display" in template
     assert "employee.last_seen_display" in template

tests/test_api_mcp.py

@@ -20,7 +20,7 @@ def test_health_returns_versions():
     response = client.get("/api/health")
 
     assert response.status_code == 200
-    assert response.json()["backend_version"] == "0.7.1"
+    assert response.json()["backend_version"] == "0.7.3"
 
 
 def test_mcp_lists_tools_without_auth_and_ignores_auth_header():
@@ -154,7 +154,7 @@ def test_mcp_service_info_returns_tools_and_dataset_hash():
     assert response.status_code == 200
     payload = json.loads(response.json()["result"]["content"][0]["text"])
     assert payload["service_name"] == "miem-employees"
-    assert payload["backend_version"] == "0.7.1"
+    assert payload["backend_version"] == "0.7.3"
     assert payload["dataset"]["hash"]
     assert any(tool["name"] == "sync_employees" for tool in payload["tools"])
 

tests/test_crawler.py

@@ -273,7 +273,7 @@ def test_upsert_employee_increments_new_count_and_records_change_for_new_employe
     db_session.add(run)
     db_session.commit()
 
-    _upsert_employee(
+    employee, _ = _upsert_employee(
         db_session,
         run,
         {
@@ -282,14 +282,20 @@ def test_upsert_employee_increments_new_count_and_records_change_for_new_employe
             "profile_id": "newperson",
             "full_name": "New Person",
             "tabs": [],
-            "sections": [],
+            "sections": [
+                {
+                    "title": "Образование и учёные степени",
+                    "year_entries": [{"text": "Кандидат технических наук"}],
+                }
+            ],
             "parser_version": "0.2.0",
             "_html": "<html></html>",
         },
     )
     db_session.commit()
 
-    assert run.new_count == 1
+    assert run.new_count == 1
+    assert employee.has_academic_degree is True
     change = db_session.query(CrawlRunEmployeeChange).one()
     assert change.change_type == "new"
     assert change.full_name == "New Person"

tests/test_db_schema.py

@@ -113,3 +113,32 @@ def test_runtime_schema_creates_employee_news_links_table_when_employees_exist(m
     assert "employee_news_links" in inspector.get_table_names()
     columns = {column["name"] for column in inspector.get_columns("employee_news_links")}
     assert {"employee_id", "title", "url", "summary", "published_at", "published_year", "source_hash", "raw_data"}.issubset(columns)
+
+
+def test_runtime_schema_adds_profile_verification_fields(monkeypatch):
+    engine = create_engine("sqlite:///:memory:")
+    with engine.begin() as connection:
+        connection.execute(
+            text(
+                """
+                CREATE TABLE employees (
+                  id INTEGER PRIMARY KEY,
+                  profile_key VARCHAR(255) NOT NULL UNIQUE,
+                  canonical_url TEXT NOT NULL,
+                  status VARCHAR(32) NOT NULL DEFAULT 'active',
+                  first_seen_at DATETIME NOT NULL,
+                  last_seen_at DATETIME NOT NULL,
+                  created_at DATETIME NOT NULL,
+                  updated_at DATETIME NOT NULL
+                )
+                """
+            )
+        )
+    monkeypatch.setattr("app.db.engine", engine)
+
+    _ensure_runtime_schema()
+
+    inspector = inspect(engine)
+    columns = {column["name"] for column in inspector.get_columns("employees")}
+    assert {"profile_unavailable_streak", "last_profile_check_at", "has_academic_degree"}.issubset(columns)
+    assert "employee_profile_urls" in inspector.get_table_names()