Compare commits

...
Author SHA1 Message Date
codex 65d851bb92 Parse mixed education records and certificates, flag unresolved DPO
CI / test (pull_request) Successful in 12m17s
CI / Fingerprint diagnostics (production deployment temporarily disabled) (pull_request) Skipped
2026-11-20 16:02:16 +00:00
agent 8dfd700bd5 Merge pull request 'Показывать загрузку и блокировать фильтры справочника' (#80) from fix/directory-loading-state into main
CI / test (push) Successful in 12m21s
CI / Fingerprint diagnostics (production deployment temporarily disabled) (push) Skipped
2026-11-20 09:11:13 +00:00
codex 2ec1ad828f docs: record directory loading fix in 2.2.1 changelog
CI / test (pull_request) Successful in 12m15s
CI / Fingerprint diagnostics (production deployment temporarily disabled) (pull_request) Skipped
2026-11-20 11:58:05 +03:00
codex 7f268abde1 fix: show directory loading and lock filter controls
CI / test (pull_request) Successful in 12m6s
CI / Fingerprint diagnostics (production deployment temporarily disabled) (pull_request) Skipped
2026-11-20 11:44:35 +03:00
agent fef738ee27 Merge pull request 'Автопоиск и основные фильтры справочника' (#79) from fix/directory-filter-headings into main
CI / test (push) Successful in 11m4s
CI / Fingerprint diagnostics (production deployment temporarily disabled) (push) Skipped
2026-11-16 15:11:45 +00:00
codex 832a8a71b6 Clarify that XLSX includes all matching rows
CI / test (pull_request) Successful in 11m15s
CI / Fingerprint diagnostics (production deployment temporarily disabled) (pull_request) Skipped
2026-11-16 18:08:49 +03:00
codex 5f6997eef8 Document automatic directory filters in release notes
CI / test (pull_request) Successful in 11m6s
CI / Fingerprint diagnostics (production deployment temporarily disabled) (pull_request) Skipped
2026-11-16 18:06:57 +03:00
codex e0599b946a Refresh employee results from primary filters automatically
CI / test (pull_request) Successful in 11m12s
CI / Fingerprint diagnostics (production deployment temporarily disabled) (pull_request) Skipped
2026-11-16 17:52:44 +03:00
codex 8d332c7d60 Remove directory filter group headings
CI / Fingerprint diagnostics (production deployment temporarily disabled) (pull_request) Canceled after 0s
CI / test (pull_request) Canceled after 9m21s
2026-11-16 17:44:46 +03:00
agent 4fd6ed72e3 Merge pull request 'Фильтры и выгрузка XLSX справочника сотрудников' (#78) from feature/directory-filters-xlsx into main
CI / test (push) Successful in 11m6s
CI / Fingerprint diagnostics (production deployment temporarily disabled) (push) Skipped
2026-11-16 13:05:42 +00:00
codex 6ad8fcd9d2 Document directory filters and XLSX export release
CI / test (pull_request) Successful in 11m8s
CI / Fingerprint diagnostics (production deployment temporarily disabled) (pull_request) Skipped
2026-11-16 16:03:33 +03:00
codex a4b0b50a0a Reveal invalid fields in collapsed directory filters
CI / test (pull_request) Successful in 11m10s
CI / Fingerprint diagnostics (production deployment temporarily disabled) (pull_request) Skipped
2026-11-16 15:44:51 +03:00
codex 1b58ab6b77 Improve directory filters and add XLSX export
CI / Fingerprint diagnostics (production deployment temporarily disabled) (pull_request) Canceled after 0s
CI / test (pull_request) Canceled after 4m58s
2026-11-16 15:41:20 +03:00
agent d1854ed5e2 Merge pull request 'Добавить роль пользователя диссовета и ограничить доступ к запускам' (#77) from feature/council-user-role into main
CI / test (push) Successful in 10m45s
CI / Fingerprint diagnostics (production deployment temporarily disabled) (push) Skipped
2026-11-12 11:35:13 +00:00
28 changed files with 1075 additions and 207 deletions

No files matched your search

+19
View File
@@ -1,5 +1,24 @@
# Changelog
## 2.3.0
- Раздел ДПО объединяет сведения из профиля и документов. Справочник показывает отдельный счётчик ДПО, а предупреждение ведёт к нераспознанным фрагментам в карточке.
## 2.2.1
- При применении фильтров справочника, переключении ППС, сортировки и размера страницы появляется заглушка «Обновляем список…». Поля, кнопки и ссылки справочника заблокированы до завершения запроса.
- Живой поиск показывает «Идёт поиск…» и оставляет управление доступным. Ошибка или тайм-аут снимают блокировку и сохраняют предыдущие результаты; устаревшие ответы не заменяют новый список.
## 2.2.0
- Поиск по ФИО или ссылке обновляет список с четвёртого символа. ППС, сортировка и размер страницы применяются сразу. XLSX выгружает все найденные строки с применёнными фильтрами и сортировкой.
- Кнопка «Применить фильтры» осталась внутри дополнительных фильтров. Из панели убраны видимые заголовки «Статус и данные», «Повышение квалификации» и «Период обнаружения».
## 2.1.1
- В справочнике всегда видны поиск по ФИО или ссылке, чекбокс «ППС», поле сортировки, стрелки направления и число сотрудников на странице. Остальные условия находятся в сворачиваемой панели.
- Кнопка «Скачать XLSX» выгружает все 15 колонок и все найденные строки со всех страниц с учётом применённых фильтров и выбранного порядка.
## 2.0.1
- Роль «Пользователь диссовета» добавляет раздел «Диссовет» к меню обычного пользователя и разрешает редактировать публикации. Без входа доступен только просмотр диссовета, включая публикации и выгрузку XLSX.
+1 -1
View File
@@ -6,7 +6,7 @@ ENV PYTHONDONTWRITEBYTECODE=1 \
WORKDIR /app
RUN apt-get update \
&& apt-get install -y --no-install-recommends gcc libpq-dev \
&& apt-get install -y --no-install-recommends gcc libpq-dev poppler-utils tesseract-ocr tesseract-ocr-rus \
&& rm -rf /var/lib/apt/lists/*
COPY requirements.txt .
+1 -1
View File
@@ -47,7 +47,7 @@ uvicorn app.main:app --reload
- «Сотрудники»: поиск, фильтры, сортировка, пагинация и выбор колонок. По умолчанию показаны ФИО, статус, должности, дата последнего обнаружения и внешний профиль. В диалоге колонок доступны наборы «Проверка», «Контакты» и «Все поля»; ранее сохранённый выбор сохраняется.
- «Запуски»: история обходов, ошибки и доступный индикатор прогресса.
Все фильтры применяются кнопкой «Применить фильтры» с переходом на первую страницу. «Сбросить» очищает условия; при устаревшем номере страницы каталог возвращает первую. Подписи полей показывают выбранные условия, над таблицей указан диапазон результатов. Пояснения дат и статусов находятся под фильтрами.
Поиск по ФИО или ссылке срабатывает с четвёртого символа. ППС, сортировка и число строк применяются сразу. Остальные условия находятся в «Дополнительных фильтрах» и применяются кнопкой внутри панели. Изменение условий возвращает на первую страницу, «Сбросить» очищает условия. «Скачать XLSX» выгружает все колонки и все строки, найденные с применёнными условиями, в выбранном порядке. Пояснения дат и статусов находятся под фильтрами.
Каталог поддерживает навигацию с клавиатуры и системный диалог колонок с Escape и возвратом фокуса. На узких экранах таблица прокручивается внутри страницы, колонка ФИО закреплена. Используется системный шрифт без внешних загрузок.
+64 -51
View File
@@ -1,5 +1,5 @@
from copy import deepcopy
from typing import Literal
from typing import Annotated, Literal
from pydantic import BaseModel, Field, ConfigDict
from fastapi import APIRouter, BackgroundTasks, Depends, Form, Query, Request, HTTPException
from fastapi.responses import HTMLResponse, RedirectResponse, Response
@@ -15,6 +15,8 @@ from app.security import (SESSION_COOKIE, LOGIN_CSRF_COOKIE, SESSION_SECONDS, US
optional_user, sign_session, authenticate, can_use_council, check_csrf)
import secrets
from app.services.admin_data import (
DIRECTORY_COLUMNS,
EMPLOYEE_SORTS,
employee_detail_payload,
format_admin_datetime,
list_employees_page,
@@ -26,6 +28,7 @@ from app.services.crawl_control import get_running_run, run_crawl_if_idle
from app.services.crawler import refresh_dismissed_status, refresh_employee
from app.services.council import CouncilInputError, council_candidates, _journal_key
from app.services.council_export import council_xlsx
from app.services.directory_export import directory_xlsx
from app.version import BACKEND_VERSION, FRONTEND_VERSION
router = APIRouter(prefix="/admin", dependencies=[Depends(check_csrf)])
@@ -109,67 +112,64 @@ def employees(
return RedirectResponse("/admin/directory", status_code=303)
@router.get("/directory", response_class=HTMLResponse)
def directory(
request: Request,
status: str | None = None,
q: str | None = None,
started_from: str | None = None,
started_to: str | None = None,
has_email: str | None = None,
has_academic_degree: str | None = None,
pps: bool = False,
has_training: Literal["", "true", "false"] = "",
training_since_year: str = Query(default="", pattern=r"^([1-9][0-9]{3})?$"),
sort: str = "full_name",
direction: str = "asc",
limit: int = 50,
offset: int = 0,
db: Session = Depends(get_db),
settings: Settings = Depends(get_settings),
):
require_user(request, settings, db)
parsed_started_from = _parse_date(started_from)
parsed_started_to = _parse_date(started_to)
parsed_has_email = None if has_email in (None, "") else has_email == "true"
parsed_has_academic_degree = None if has_academic_degree in (None, "") else has_academic_degree == "true"
try:
page = list_employees_page(
class DirectoryFilters(BaseModel):
status: str = ""
q: str = ""
started_from: str = ""
started_to: str = ""
has_email: str = ""
has_academic_degree: str = ""
pps: bool = False
has_training: Literal["", "true", "false"] = ""
training_since_year: str = Field(default="", pattern=r"^([1-9][0-9]{3})?$")
sort: str = "full_name"
direction: str = "asc"
limit: int = 50
offset: int = 0
def employee_page(self, db: Session, *, paginate: bool = True) -> dict:
return list_employees_page(
db,
status=status,
q=q,
started_from=parsed_started_from,
started_to=parsed_started_to,
has_email=parsed_has_email,
has_academic_degree=parsed_has_academic_degree,
pps=pps,
has_training=has_training == "true" if has_training else None,
training_since_year=int(training_since_year) if training_since_year else None,
sort=sort,
direction=direction,
limit=limit,
offset=offset,
status=self.status,
q=self.q,
started_from=_parse_date(self.started_from),
started_to=_parse_date(self.started_to),
has_email=self.has_email == "true" if self.has_email else None,
has_academic_degree=self.has_academic_degree == "true" if self.has_academic_degree else None,
pps=self.pps,
has_training=self.has_training == "true" if self.has_training else None,
training_since_year=int(self.training_since_year) if self.training_since_year else None,
sort=self.sort,
direction=self.direction,
limit=self.limit,
offset=self.offset,
paginate=paginate,
)
@router.get("/directory", response_class=HTMLResponse)
def directory(request: Request, filters: Annotated[DirectoryFilters, Query()],
db: Session = Depends(get_db), settings: Settings = Depends(get_settings)):
require_user(request, settings, db)
try:
page = filters.employee_page(db)
except SQLAlchemyError:
db.rollback()
return _render(request, "directory_error.html", {}, status_code=503)
export_url = request.url_for("directory_export").include_query_params(
**filters.model_dump(exclude={"limit", "offset"}, exclude_defaults=True)
)
return _render(
request,
"directory.html",
{
"page": page,
"directory_columns": DIRECTORY_COLUMNS,
"export_url": export_url,
"filters": {
"status": status or "",
"q": q or "",
"started_from": started_from or "",
"started_to": started_to or "",
"has_email": has_email or "",
"has_academic_degree": has_academic_degree or "",
"pps": pps,
"has_training": has_training,
"training_since_year": training_since_year,
"sort": sort,
"direction": direction,
**filters.model_dump(),
"sort": filters.sort if filters.sort in EMPLOYEE_SORTS else "full_name",
"direction": "desc" if filters.direction == "desc" else "asc",
"limit": page["limit"],
"offset": page["offset"],
},
@@ -177,6 +177,19 @@ def directory(
)
@router.get("/directory/export.xlsx")
def directory_export(request: Request, filters: Annotated[DirectoryFilters, Query()],
db: Session = Depends(get_db), settings: Settings = Depends(get_settings)):
require_user(request, settings, db)
try:
content = directory_xlsx(filters.employee_page(db, paginate=False)["employees"])
except (SQLAlchemyError, OSError, ValueError):
db.rollback()
raise HTTPException(503, "Не удалось выгрузить сотрудников. Попробуйте ещё раз")
return Response(content, media_type="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
headers={"Content-Disposition": 'attachment; filename="employees.xlsx"', "Cache-Control": "no-store"})
@router.get("/council", response_class=HTMLResponse)
def council(request: Request, q: str = "", offset: int = 0, sort: str = "total_count", direction: str = "desc", specialty: str = "", branch: str = "технические",
db: Session = Depends(get_db), settings: Settings = Depends(get_settings)):
+65
View File
@@ -0,0 +1,65 @@
"""Assemble additional education events from the public profile's mixed markup."""
import re
YEAR = re.compile(r"\b(?:19|20)\d{2}\b")
_QUOTE = re.compile(r'«([^»]+)»|"([^"]+)"')
_EVENT = re.compile(r"стажиров|стипенди|исследовательск|обучени|квалификаци|переподготовк|прош[её]л|прошла|семинар|сертификаци|программ[аеуы]|курсы?\b", re.I)
_CONTINUES = re.compile(r"(?:по программе|присвоением степени|на тему|по курсу|по теме|по специальности)[:\s]*$", re.I)
_CAPTION = re.compile(r"^(?:удостоверение|сертификат|свидетельство)(?: о повышении квалификации)?[.:]?$", re.I)
def _key(text):
return " ".join(re.findall(r"\w+", text.casefold().replace("ё", "е")))
def _provider(text):
text = _key(text)
for variants in (("вшэ", "высшая школа экономики", "высшей школе экономики"),
("мифи", "московский инженерно физический институт")):
if any(v in text.split() if " " not in v else v in text for v in variants):
return variants[0]
return text
def assemble(rows):
events = []
unresolved = []
pending = []
for text, year, structured, list_item in rows:
if _CAPTION.fullmatch(text):
continue
if (pending and not structured and not list_item and not YEAR.match(text)
and (_CONTINUES.search(pending[-1][0]) or
(pending[-1][3] and not _EVENT.search(text[:20]) and not YEAR.search(text)) or
(pending[-1][3] and text.startswith("Курс ") and not YEAR.search(text)))):
pending[-1][0] += " " + text
pending[-1][3] = False
else:
pending.append([text, year, structured, list_item])
for text, year, structured, _ in pending:
if not structured and (len(_key(text).split()) < 2 or not (_EVENT.search(text) or YEAR.match(text))):
unresolved.append(text)
continue
quotes = [a or b for a, b in _QUOTE.findall(text)]
title = _key(quotes[-1]) if quotes else ""
if structured and text.lstrip("0123456789 ").startswith("«"):
title = _key(text.split("«", 1)[1].rsplit("»", 1)[0])
provider = _provider(text.rsplit("»", 1)[-1] if structured and "»" in text else text)
if not year:
match = YEAR.search(_QUOTE.sub("", text))
year = int(match[0]) if match else None
matches = [e for e in events if title and e["title"] == title and e["provider"] == provider
and (e["year"] == year or not year or not e["year"])]
if len(matches) == 1:
target = matches[0]
hours = re.search(r"\b\d+\s+час[а-я]*", text, re.I)
if hours and hours[0] not in target["text"]:
target["text"] += " (" + hours[0] + ")"
continue
if any(e["text"] == text for e in events):
continue
events.append(dict(text=text, year=year, structured=structured, title=title, provider=provider))
return events, unresolved
+70
View File
@@ -0,0 +1,70 @@
"""Read public HSE education documents. An inconclusive extraction stays visible for review."""
import hashlib
import re
import subprocess
from pathlib import Path
from tempfile import TemporaryDirectory
from urllib.parse import urlsplit
import requests
_MAX_BYTES = 12 * 1024 * 1024
def _run(args):
return subprocess.run(args, check=True, capture_output=True, timeout=30).stdout.decode("utf-8")
def _certificate(text):
text = " ".join(text.split())
title = re.search(r'(?:по (?:дополнительной профессиональной )?программе|по теме|на тему)\s*[:«"\s]*([^»"]{5,300})[»"]', text, re.I)
if not title:
title = re.search(r'[«"]([^«»"]{8,250})[»"]\s*[,.(]?\s*(?:в объ[её]ме|\d+ час)', text, re.I)
years = re.findall(r"\b(?:19|20)\d{2}\b", text)
if not title or not years or not re.search(r"квалификаци|сертификат|удостоверение", text, re.I):
return None
hours = re.search(r"(\d+)\s*(?:академических\s+)?час", text, re.I)
provider = ""
if re.search(r"высш[а-я]+ школ[а-я]+ экономик|НИУ ВШЭ", text, re.I):
provider = ", НИУ ВШЭ"
elif re.search("МИСИС", text, re.I):
provider = ", НИТУ МИСИС"
return f'{years[-1]} «{title[1].strip()}», повышение квалификации{provider}' + (f", {hours[1]} часов" if hours else "")
def _recognize(data):
with TemporaryDirectory() as tmp:
src = Path(tmp) / "document"
src.write_bytes(data)
images = [src]
if data.startswith(b"%PDF"):
direct = _certificate(_run(["pdftotext", "-layout", str(src), "-"]))
if direct:
return direct
_run(["pdftoppm", "-f", "1", "-l", "2", "-scale-to", "2400", "-png", str(src), str(src)])
images = sorted(Path(tmp).glob("document-*.png"))
elif not data.startswith((b"\x89PNG", b"\xff\xd8\xff")):
return None
return _certificate(" ".join(_run(["tesseract", str(img), "stdout", "-l", "rus+eng"]) for img in images))
def read_document(session, url, timeout, cache, profile_key):
parts = urlsplit(url)
if parts.scheme != "https" or parts.netloc != "www.hse.ru" or not parts.path.startswith("/mirror/pubs/share/"):
return None
try:
with session.get(url, stream=True, timeout=timeout, allow_redirects=False) as response:
if response.status_code != 200:
return None
data = bytearray()
for chunk in response.iter_content(65536):
data.extend(chunk)
if len(data) > _MAX_BYTES:
return None
digest = hashlib.sha256(data).hexdigest()
if cache:
return cache.education_document(profile_key, url, digest, lambda: _recognize(data))
return _recognize(data)
except (OSError, ValueError, subprocess.SubprocessError, requests.RequestException):
return None
+81 -3
View File
@@ -1,12 +1,15 @@
import hashlib
import json
import re
from copy import deepcopy
from datetime import datetime, timezone
from urllib.parse import urljoin
from bs4 import BeautifulSoup, NavigableString, Tag
from bs4 import BeautifulSoup, Comment, NavigableString, Tag
from requests import Session
from app.parser.education import assemble
from app.parser.education_document import read_document
from app.parser.profile_url import normalize_profile_url, parse_profile_identity
from app.version import BACKEND_VERSION
@@ -99,7 +102,7 @@ def extract_person_header(soup: BeautifulSoup, source_url: str) -> dict:
}
def extract_sections(soup: BeautifulSoup, source_url: str) -> list[dict]:
def extract_sections(soup: BeautifulSoup, source_url: str, document_reader=None) -> list[dict]:
sections = []
for h2 in soup.select("h2"):
if h2.find_parent(class_="post") or h2.find_parent(attrs={"data-tab": "press_links_news"}):
@@ -108,6 +111,11 @@ def extract_sections(soup: BeautifulSoup, source_url: str) -> list[dict]:
if not title or "расписание занятий" in title.lower():
continue
nodes = _collect_between_h2(h2)
education_titles = {part.strip().rstrip(":") for part in title.casefold().split("/")}
if (h2.find_parent(attrs={"tab-node": "additional_education"})
or education_titles & {"дополнительное образование", "повышение квалификации", "курсы повышения квалификации"}):
sections.append(_parse_additional_education(title, nodes, source_url, document_reader))
continue
raw_text = _nodes_raw_text(nodes)
paragraphs = _nodes_paragraphs(nodes)
items = _nodes_list_items(nodes)
@@ -193,7 +201,7 @@ def parse_person_profile(
soup = BeautifulSoup(html, "html.parser")
header = extract_person_header(soup, normalized_url)
tabs = extract_person_tabs(soup, normalized_url)
sections = extract_sections(soup, normalized_url)
sections = extract_sections(soup, normalized_url, lambda url: read_document(session, url, timeout, resource_cache, cache_profile_key))
sections = enrich_sections_from_hse_widgets(
session,
soup,
@@ -219,6 +227,7 @@ def parse_person_profile(
"tabs": tabs,
"sections": sections,
"employee_internal_links": internal_links,
"education_parsed": True,
"parser_version": BACKEND_VERSION,
"_html": html,
"_resource_manifest": resource_manifest,
@@ -585,6 +594,75 @@ def _parse_year_entries(nodes: list, source_url: str) -> list[dict]:
return entries
_FILE_FORMAT = r"(?:pdf|jpe?g|png|docx?|pptx?|xlsx?|zip|rar)"
_FILE_SIZE = re.compile(rf"\(\s*{_FILE_FORMAT}\s*,\s*[\d.,\s]+[кмгkmg]?\s*[бb](?:айт[а-я]*)?\s*\)", re.I)
_FILE_CAPTION = re.compile(rf"\b(?:файл|сертификат|свидетельство|удостоверение)(?:_\w+)?\s*{_FILE_SIZE.pattern}", re.I)
_FILE_URL = re.compile(rf"/mirror/pubs/share/|\.{_FILE_FORMAT}(?:[?#]|$)", re.I)
_INITIAL_YEAR = re.compile(r"^(?:19|20)\d{2}\b")
def _education_blocks(node):
"""Yield whole course rows or paragraphs once, in document order."""
if not isinstance(node, Tag):
if not isinstance(node, Comment) and normalize_ws(str(node)):
yield node
return
if node.name == "tr" and node.find("th"):
return
if (node.name in {"li", "p", "tr"} or node.find(class_="person-list-hangover", recursive=False)
or not node.find(["div", "ul", "ol", "li", "p", "table", "tr"])):
yield node
return
for child in node.children:
yield from _education_blocks(child)
def _parse_additional_education(title: str, nodes: list, source_url: str, document_reader=None) -> dict:
rows = []
documents = {}
unresolved_documents = []
links = []
for original in nodes:
node = deepcopy(original)
if isinstance(node, Tag):
for anchor in node.select("a[href]"):
following = anchor.next_sibling
suffix = following.get_text(strip=True) if isinstance(following, Tag) else str(following or "")
if (_FILE_URL.search(anchor["href"]) or anchor.has_attr("download")
or _FILE_SIZE.match(normalize_ws(suffix))):
documents[urljoin(source_url, anchor["href"])] = normalize_ws(anchor.get_text(" ", strip=True)) or "Документ"
anchor.decompose()
for block in _education_blocks(node):
is_tag = isinstance(block, Tag)
content = block.get_text(" ", strip=True) if is_tag else str(block)
text = normalize_ws(_FILE_SIZE.sub("", _FILE_CAPTION.sub("", content)))
text = re.sub(r"\s+([,.;:])", r"\1", text)
text = re.sub(r"(?<=[.!?])\.$", "", text)
if not re.search(r"\w", text):
continue
block_links = _extract_links(block, source_url) if is_tag else []
links.extend(block_links)
year_node = block.find(class_="person-list-hangover", recursive=False) if is_tag else None
year_match = _INITIAL_YEAR.match(year_node.get_text(strip=True)) if year_node else None
rows.append((text, int(year_match[0]) if year_match else None, bool(year_node), is_tag and block.name == "li"))
for url, caption in documents.items():
recognized = document_reader(url) if document_reader else None
if recognized:
rows.append((recognized, None, False, False))
elif document_reader:
unresolved_documents.append({"text": caption, "url": url})
events, unresolved = assemble(rows)
items = [event["text"] for event in events]
year_entries = [{"year": event["year"], "text": event["text"], "links": []}
for event in events if event["structured"]]
return {
"title": title, "slug": _slugify(title), "type": "additional_education",
"items": items, "paragraphs": [], "year_entries": year_entries,
"unresolved": unresolved, "unresolved_documents": unresolved_documents,
"raw_text": "\n".join(items), "links": links,
}
def _parse_vkr_items(nodes: list) -> list[str]:
items = []
for node in nodes:
+43 -5
View File
@@ -7,7 +7,7 @@ from typing import Any
from zoneinfo import ZoneInfo
from sqlalchemy import Select, Text, and_, desc, func, or_, select
from sqlalchemy.orm import Session
from sqlalchemy.orm import Session, selectinload
from app.models import CrawlError, CrawlRun, CrawlRunEmployeeChange, Employee, EmployeeNewsLink
from app.services.academic_degrees import academic_degrees
@@ -21,6 +21,26 @@ EMPLOYEE_SORTS = {
"hse_start_year": Employee.current_data["hse_start_year"].as_integer(),
}
# Table key, column title, and the corresponding value in the display payload.
DIRECTORY_COLUMNS = (
("full_name", "ФИО", "full_name"),
("status", "Статус", "status_display"),
("positions", "Должности", "positions_text"),
("hse_start_year", "Год начала", "hse_start_year"),
("email", "Email", "email_text"),
("phone", "Телефон", "phone_text"),
("address", "Адрес", "address"),
("academic_degree", "Учёная степень", "academic_degree_text"),
("publications_count", "Публикации", "publications_count"),
("courses_count", "Курсы", "courses_count"),
("news_count", "Новости", "news_count"),
("first_seen_at", "Впервые найден", "first_seen_display"),
("last_seen_at", "Последний раз найден", "last_seen_display"),
("dismissed_at", "Дата увольнения", "dismissed_display"),
("profile", "Профиль", "canonical_url"),
("dpo_count", "Количество ДПО", "dpo_count"),
)
_TRAINING_LABEL = "повышение квалификации"
_TRAINING_KIND = re.compile(r",\s*повышение квалификации\s*(?:,|$)", re.IGNORECASE)
_TRAINING_MENTION = re.compile(
@@ -38,16 +58,20 @@ def _has_training(data: dict[str, Any], since_year: int | None) -> bool:
dedicated = title in {_TRAINING_LABEL, "курсы повышения квалификации"}
if not dedicated and _TRAINING_LABEL not in (part.strip() for part in title.split("/")):
continue
structured_texts = set()
for value in _as_list(section.get("year_entries")):
entry = _as_dict(value)
year = entry.get("year")
text = str(entry.get("text") or "").strip()
structured_texts.add(text)
if since_year is not None and (type(year) is not int or year < since_year):
continue
if text and text != str(year) and (dedicated or _TRAINING_KIND.search(_QUOTED_TEXT.sub("", text))):
return True
# Profiles may also list completed courses in manually entered paragraphs or lists.
for text in _clean_list(section.get("paragraphs")) + _clean_list(section.get("items")):
if text in structured_texts:
continue
description = _QUOTED_TEXT.sub("", text).strip()
if not dedicated and not _TRAINING_MENTION.search(description):
continue
@@ -68,6 +92,8 @@ def employee_display_payload(employee: Employee) -> dict[str, Any]:
emails = _clean_list(contacts.get("emails"))
phones = _clean_list(contacts.get("phones"))
degree_values = academic_degrees(data)
education = [s for s in sections if isinstance(s, dict) and s.get("type") == "additional_education"]
parsed_education = bool(education) or data.get("education_parsed", False)
return {
"id": employee.id,
"full_name": employee.full_name,
@@ -84,6 +110,8 @@ def employee_display_payload(employee: Employee) -> dict[str, Any]:
"address": contacts.get("address"),
"academic_degree_text": "; ".join(degree_values),
"publications_count": _count_section_items(sections, "publications"),
"dpo_count": sum(len(s.get("items", [])) for s in education) if parsed_education else None,
"dpo_uncertain": any(s.get("unresolved") or s.get("unresolved_documents") for s in education),
"courses_count": _count_section_items(sections, "courses_by_year"),
"news_count": len(stored_news_links) or _count_section_items(sections, "news"),
"first_seen_at": employee.first_seen_at.isoformat() if employee.first_seen_at else None,
@@ -162,6 +190,7 @@ def list_employees_page(
direction: str = "asc",
limit: int = 50,
offset: int = 0,
paginate: bool = True,
) -> dict[str, Any]:
limit = limit if limit in {25, 50, 100} else 50
offset = max(0, offset)
@@ -175,7 +204,9 @@ def list_employees_page(
)
sort_column = EMPLOYEE_SORTS.get(sort, Employee.full_name)
order = desc(sort_column) if direction == "desc" else sort_column
ordered_stmt = base_stmt.order_by(order)
ordered_stmt = base_stmt.order_by(order, Employee.id)
if not paginate:
ordered_stmt = ordered_stmt.options(selectinload(Employee.news_links))
if has_training is None and training_since_year is not None:
has_training = True
if pps or has_training is not None:
@@ -190,19 +221,24 @@ def list_employees_page(
continue
if has_training is not None and _has_training(data, training_since_year) != has_training:
continue
if offset and total < limit:
if paginate and offset and total < limit:
first_page.append(employee)
if offset <= total < offset + limit:
if not paginate or offset <= total < offset + limit:
employees.append(employee)
total += 1
if offset >= total:
if paginate and offset >= total:
employees = first_page
offset = 0
elif not paginate:
employees = db.scalars(ordered_stmt).all()
total = len(employees)
else:
total = db.scalar(select(func.count()).select_from(base_stmt.subquery())) or 0
if offset >= total:
offset = 0
employees = db.scalars(ordered_stmt.limit(limit).offset(offset)).all()
if not paginate:
offset = 0
return {
"employees": [employee_display_payload(employee) for employee in employees],
"total": total,
@@ -429,6 +465,8 @@ def _normalize_section(section: Any) -> dict[str, Any]:
"raw_text": raw_text,
"paragraphs": paragraphs,
"list_items": items,
"unresolved": _clean_list(section.get("unresolved")),
"unresolved_documents": _normalize_links(section.get("unresolved_documents")),
"links": _normalize_links(section.get("links")),
"year_entries": _normalize_year_entries(section.get("year_entries")),
"publications": _normalize_publications(section.get("publications")),
+1 -12
View File
@@ -4,7 +4,6 @@ from io import BytesIO
from textwrap import fill
from openpyxl import Workbook
from openpyxl.cell.cell import ILLEGAL_CHARACTERS_RE
from openpyxl.styles import Alignment, Font, PatternFill
from openpyxl.utils import get_column_letter
from openpyxl.worksheet.datavalidation import DataValidation
@@ -13,23 +12,13 @@ from openpyxl.workbook.defined_name import DefinedName
from app.version import BACKEND_VERSION, FRONTEND_VERSION
from app.services.council_source import _name
from app.services.xlsx import append_text_row as _append
SORT_LABELS = {"full_name": "ФИО", "q1_count": "Q1", "q2_count": "Q2", "k1_count": "К1",
"k2_count": "К2", "rsci_count": "RSCI", "total_count": "Всего",
"confirmed_count": "Подтверждено", "db_count": "ЛК сотрудника ВШЭ", "count": "Все публикации"}
def _append(sheet, values):
values = [ILLEGAL_CHARACTERS_RE.sub("", value) if isinstance(value, str) else value for value in values]
if any(isinstance(value, str) and len(value) > 32767 for value in values):
raise ValueError("Text exceeds Excel cell limit")
sheet.append(values)
for column, value in enumerate(values, 1):
if isinstance(value, str):
# Source text is never executable; formulas are written separately.
sheet.cell(sheet._current_row, column).data_type = "s"
def council_xlsx(result: dict, selection_results: list[dict]) -> bytes:
workbook = Workbook()
workbook.properties.creator = "MIEM Employees"
+29
View File
@@ -0,0 +1,29 @@
from io import BytesIO
from openpyxl import Workbook
from openpyxl.styles import Font
from openpyxl.utils import get_column_letter
from app.services.admin_data import DIRECTORY_COLUMNS
from app.services.xlsx import append_text_row
def directory_xlsx(employees: list[dict]) -> bytes:
workbook = Workbook()
workbook.properties.creator = "MIEM Employees"
sheet = workbook.active
sheet.title = "Сотрудники"
append_text_row(sheet, [label for _, label, _ in DIRECTORY_COLUMNS])
for employee in employees:
append_text_row(sheet, [
(employee[key] or "Без имени") if key == "full_name" else employee[key]
for _, _, key in DIRECTORY_COLUMNS
])
for cell, width in zip(sheet[1], (34, 20, 48, 14, 28, 22, 34, 30, 14, 10, 10, 22, 24, 22, 46, 18)):
cell.font = Font(bold=True)
sheet.column_dimensions[get_column_letter(cell.column)].width = width
sheet.freeze_panes = "A2"
sheet.auto_filter.ref = sheet.dimensions
output = BytesIO()
workbook.save(output)
return output.getvalue()
+18
View File
@@ -27,6 +27,24 @@ class ResourceCache:
def __init__(self, db: Session):
self.db = db
def education_document(self, profile_key, url, digest, recognize):
key = "education-document-v1"
existing = self.db.scalar(select(ParseResourceCache).where(
ParseResourceCache.profile_key == profile_key, ParseResourceCache.resource_key == key,
ParseResourceCache.request_fingerprint == digest))
if existing:
return gzip.decompress(existing.body_snapshot).decode() or None
result = recognize()
from sqlalchemy.exc import IntegrityError
try:
with self.db.begin_nested():
self.db.add(ParseResourceCache(profile_key=profile_key, resource_key=key, method="OCR", url=url,
request_fingerprint=digest, body_hash=digest, body_snapshot=gzip.compress((result or "").encode()),
parser_version=BACKEND_VERSION))
except IntegrityError:
pass # Another crawl cached the same document.
return result
def fetch_text(
self,
session: requests.Session,
+12
View File
@@ -0,0 +1,12 @@
from openpyxl.cell.cell import ILLEGAL_CHARACTERS_RE
def append_text_row(sheet, values):
values = [ILLEGAL_CHARACTERS_RE.sub("", value) if isinstance(value, str) else value for value in values]
if any(isinstance(value, str) and len(value) > 32767 for value in values):
raise ValueError("Text exceeds Excel cell limit")
sheet.append(values)
for column, value in enumerate(values, 1):
if isinstance(value, str):
# Source text is never executable; formulas are written separately.
sheet.cell(sheet._current_row, column).data_type = "s"
+120 -29
View File
@@ -67,7 +67,7 @@
outline-offset: 2px;
}
:is(.directory__filters, .council__search) :is(input, select):focus {
:is(.directory__toolbar, .directory__filters, .council__search) :is(input, select):focus {
outline: 2px solid #0f766e;
outline-offset: 0;
}
@@ -564,6 +564,92 @@
color: #6b7280;
}
.directory__actions,
.directory__toolbar {
display: flex;
flex-wrap: wrap;
align-items: end;
gap: 8px;
}
.directory__toolbar {
column-gap: 12px;
}
.directory__search {
flex: 1 1 240px;
}
.directory__direction {
display: flex;
gap: 4px;
min-width: 0;
margin: 0;
padding: 0;
border: 0;
}
.directory__direction legend {
margin-bottom: 5px;
padding: 0;
color: #4b5563;
font-size: 12px;
font-weight: 700;
}
.directory__direction label {
position: relative;
cursor: pointer;
}
.directory__direction input {
position: absolute;
inline-size: 1px;
block-size: 1px;
margin: 0;
opacity: 0;
}
.directory__direction span {
display: grid;
place-items: center;
box-sizing: border-box;
width: 44px;
height: 44px;
color: #1f2937;
background: #fff;
border: 1px solid #cbd5e1;
border-radius: 6px;
}
.directory__direction label:hover span {
border-color: #0f766e;
}
.directory__direction input:checked + span {
color: #fff;
background: #0f766e;
border-color: #0f766e;
}
.directory__direction input:focus-visible + span {
outline: 2px solid #0f766e;
outline-offset: 2px;
}
@media (forced-colors: active) {
.directory__direction input:checked + span {
color: HighlightText;
background: Highlight;
border-color: Highlight;
forced-color-adjust: none;
}
}
.directory__filter-panel {
margin-top: 8px;
}
.directory__filter-panel > summary {
width: fit-content;
min-height: 44px;
@@ -574,7 +660,7 @@
.directory__filters {
display: grid;
grid-template-columns: repeat(5, minmax(0, 1fr));
grid-template-columns: repeat(3, minmax(0, 1fr));
gap: 10px;
padding: 16px;
background: #ffffff;
@@ -593,18 +679,6 @@
border: 0;
}
.directory__filter-group--search {
grid-template-columns: 1fr;
}
.directory__filter-legend {
grid-column: 1 / -1;
margin-bottom: 2px;
color: #374151;
font-size: 12px;
font-weight: 700;
}
.directory__field {
display: grid;
gap: 5px;
@@ -654,6 +728,29 @@
border-radius: 8px;
}
.directory__results {
position: relative;
min-width: 0;
}
.directory__loading:not([hidden]) {
position: absolute;
inset: 0;
z-index: 3;
display: grid;
place-items: start center;
padding: 32px 16px;
color: #1f2937;
background: rgb(255 255 255 / 92%);
border-radius: 8px;
font-weight: 700;
}
.directory--loading :is(input:disabled, select:disabled, button:disabled, a[inert], summary[inert], .directory__direction input:disabled + span) {
cursor: wait;
opacity: 0.6;
}
.directory__pagination {
display: flex;
align-items: center;
@@ -801,22 +898,11 @@
font-weight: 700;
}
@media (max-width: 1100px) {
.directory__filters {
grid-template-columns: repeat(3, minmax(0, 1fr));
}
}
@media (max-width: 920px) {
.directory__filters {
grid-template-columns: 1fr 1fr;
}
.directory__filter-group--search,
.directory__filter-actions {
grid-column: 1 / -1;
}
.progress-panel__header,
.directory__header,
.employee-card__header {
@@ -855,10 +941,12 @@
grid-template-columns: 1fr;
}
.directory__filter-group,
.directory__filter-group--search,
.directory__filter-actions {
grid-column: 1;
.directory__search {
flex-basis: 100%;
}
.directory__sort {
flex: 1 1 120px;
}
.directory__field {
@@ -1084,3 +1172,6 @@
.guide__fields dd { margin: 4px 0 0; overflow-wrap: anywhere; }
.guide--page { max-width: 75ch; }
.guide--page .guide__step + .guide__step { margin-block-start: 32px; }
.dpo-warning { color: #ab202b; text-decoration: none; }
#unresolved-dpo:target { outline: 2px solid #ab202b; outline-offset: 3px; }
+140 -1
View File
@@ -10,7 +10,7 @@
const columnPresets = {
review: ["full_name", "status", "academic_degree", "last_seen_at", "profile"],
contacts: ["full_name", "status", "email", "phone", "address", "profile"],
full: ["full_name", "status", "positions", "hse_start_year", "email", "phone", "address", "academic_degree", "publications_count", "courses_count", "news_count", "first_seen_at", "last_seen_at", "dismissed_at", "profile"],
full: ["full_name", "status", "positions", "hse_start_year", "email", "phone", "address", "academic_degree", "publications_count", "courses_count", "news_count", "first_seen_at", "last_seen_at", "dismissed_at", "profile", "dpo_count"],
};
function readColumns() {
@@ -78,6 +78,144 @@
});
}
function setupDirectoryFilters() {
const form = document.querySelector("[data-directory-filters]");
if (!form) return;
const search = form.elements.q;
const summary = document.querySelector("[data-directory-summary]");
const exportLink = document.querySelector("[data-directory-export]");
const results = document.querySelector("[data-directory-results]");
const pagination = document.querySelector("[data-directory-pagination]");
const directory = form.closest(".directory");
const overlay = directory.querySelector("[data-directory-loading]");
const fields = ["q", "pps", "sort", "direction", "limit"];
const readControls = () => {
const data = new FormData(form);
return new URLSearchParams(fields.map((name) => [name, data.get(name) || ""]));
};
let applied = readControls();
let appliedSummary = summary.textContent;
let timer;
let pending;
let lockedControls = [];
let lockedLinks = [];
function setLoading(loading, blocking = false) {
lockedControls.forEach((control) => { control.disabled = false; });
lockedLinks.forEach((link) => { link.inert = false; });
lockedControls = [];
lockedLinks = [];
overlay.hidden = !loading || !blocking;
results.toggleAttribute("inert", loading && blocking);
results.setAttribute("aria-busy", String(loading));
directory.classList.toggle("directory--loading", loading && blocking);
if (loading) {
summary.textContent = blocking ? "Обновляем список…" : "Идёт поиск…";
}
if (loading && blocking) {
lockedControls = Array.from(directory.querySelectorAll("input:not(:disabled), select:not(:disabled), button:not(:disabled)"));
lockedLinks = Array.from(directory.querySelectorAll("a:not([inert]), summary:not([inert])"));
lockedControls.forEach((control) => { control.disabled = true; });
lockedLinks.forEach((link) => { link.inert = true; });
}
}
form.querySelector(".directory__filter-panel").addEventListener("invalid", (event) => {
event.currentTarget.open = true;
}, true);
form.addEventListener("formdata", (event) => {
const text = search.value.trim();
if (Array.from(text).length >= 4) event.formData.set("q", text);
else event.formData.delete("q");
});
async function refresh(blocking = false, manual = false) {
const next = readControls();
if (!manual && next.toString() === applied.toString()) {
setLoading(false);
if (summary.textContent !== appliedSummary) summary.textContent = appliedSummary;
return;
}
// Keep applied additional filters even if their fields have unsubmitted edits.
const url = new URL(window.location.href);
url.searchParams.delete("offset");
if (manual) url.search = new URLSearchParams(new FormData(form)).toString();
for (const [name, value] of next) {
if (value) url.searchParams.set(name, value);
else url.searchParams.delete(name);
}
const request = new AbortController();
pending = request;
setLoading(true, blocking);
try {
const response = await fetch(url, {
signal: AbortSignal.any([request.signal, AbortSignal.timeout(15000)]),
});
if (!response.ok) throw new Error("directory request failed");
const page = new DOMParser().parseFromString(await response.text(), "text/html");
if (request.signal.aborted) return;
const nextSummary = page.querySelector("[data-directory-summary]");
const nextExport = page.querySelector("[data-directory-export]");
const nextResults = page.querySelector("[data-directory-results]");
const nextPagination = page.querySelector("[data-directory-pagination]");
if (!nextSummary || !nextExport || !nextResults || !nextPagination) throw new Error("directory results missing");
history.replaceState(history.state, "", url);
results.replaceChildren(...nextResults.childNodes);
pagination.replaceChildren(...nextPagination.childNodes);
applyColumns(readColumns());
exportLink.setAttribute("href", nextExport.getAttribute("href"));
appliedSummary = nextSummary.textContent;
summary.textContent = appliedSummary;
applied = next;
if (manual) {
const panel = form.querySelector(".directory__filter-panel");
panel.querySelector("summary").textContent = page.querySelector(".directory__filter-panel > summary").textContent;
panel.open = false;
}
} catch (_error) {
if (!request.signal.aborted) {
summary.textContent = "Не удалось обновить список. Показаны предыдущие результаты.";
}
} finally {
if (pending === request) {
setLoading(false);
pending = null;
}
}
}
function schedule(delay = 0, blocking = false, manual = false) {
clearTimeout(timer);
pending?.abort();
pending = null;
setLoading(false);
if (delay) {
if (readControls().toString() !== applied.toString()) setLoading(true);
else summary.textContent = appliedSummary;
timer = setTimeout(() => refresh(), delay);
} else refresh(blocking, manual);
}
form.addEventListener("submit", (event) => {
event.preventDefault();
schedule(0, true, true);
});
search.addEventListener("input", (event) => {
if (!event.isComposing) schedule(250);
});
search.addEventListener("compositionend", () => schedule(250));
search.addEventListener("keydown", (event) => {
if (event.key === "Enter" && !event.isComposing) {
event.preventDefault();
schedule();
}
});
form.querySelector(".directory__toolbar").addEventListener("change", (event) => {
if (event.target !== search) schedule(0, true);
});
}
function setupProgress() {
const panel = document.querySelector("[data-progress-panel]");
if (!panel) return;
@@ -149,5 +287,6 @@
}
setupColumns();
setupDirectoryFilters();
setupProgress();
})();
+7 -9
View File
@@ -10,24 +10,22 @@
{id: 'welcome', title: 'Познакомьтесь с Workers',
intro: (councilAccess ? 'Найдите сотрудника, подберите кандидатов в диссовет и проверьте публикации. '
: 'Найдите сотрудника и изучите его профиль. ') + 'Подсказки можно повторить из меню «Как пользоваться».'},
{id: 'employee-search', path: directory, title: 'Найдите сотрудника', region: '.directory__filter-panel', tips: [
['[name="q"]', 'Введите фамилию, часть ФИО или ссылку на профиль ВШЭ.'],
['[name="status"]', '«Требует проверки» означает, что профиль недоступен, а увольнение ещё не подтверждено.'],
['[name="has_email"]', 'Чтобы найти рабочие контакты, выберите «Есть email».'],
{id: 'employee-search', path: directory, title: 'Найдите сотрудника', region: '.directory__toolbar', tips: [
['[name="q"]', 'Введите фамилию или ссылку на профиль ВШЭ. Поиск срабатывает с четвёртого символа.'],
['[name="pps"]', 'Оставьте сотрудников профессорско-преподавательского состава.'],
]},
{id: 'employee-filters', path: directory, title: 'Уточните список', region: '.directory__filter-panel', tips: [
['[name="has_academic_degree"]', 'Звание профессора или доцента само по себе не считается учёной степенью.'],
['[name="pps"]', 'Оставьте сотрудников профессорско-преподавательского состава.'],
['[name="has_email"]', 'Чтобы найти рабочие контакты, выберите «Есть email».'],
['[name="started_to"]', '«От» и «До» задают период первого обнаружения профиля системой.'],
]},
{id: 'employee-training', path: directory, title: 'Проверьте повышение квалификации', region: '.directory__filter-panel', tips: [
['[name="has_training"]', 'Выберите «Есть курсы» или «Нет курсов» по записям в профиле ВШЭ.'],
['[name="training_since_year"]', 'Например, 2024 включает 2024 год и все последующие.'],
]},
{id: 'employee-sort', path: directory, title: 'Настройте порядок списка', region: '.directory__filter-panel', tips: [
['[name="sort"]', '«Год начала» — начало работы в ВШЭ. «Впервые найден» — когда профиль обнаружила система.'],
['[name="direction"]', 'Чтобы увидеть новые записи первыми, выберите «По убыванию».'],
['[name="limit"]', 'Выберите число строк и нажмите «Применить фильтры».'],
{id: 'employee-sort', path: directory, title: 'Настройте порядок списка', region: '.directory__toolbar', tips: [
['[name="sort"]', 'Выберите поле сортировки. Стрелка вниз покажет наибольшие значения или новые даты первыми.'],
['[name="limit"]', 'Выберите число строк. Список обновится автоматически.'],
]},
{id: 'employee-profile', path: directory, title: 'Откройте карточку сотрудника', region: '.directory__header', tips: [
['[data-columns-open]', 'Выберите столбцы: например, должности, контакты и число публикаций.'],
+83 -83
View File
@@ -7,104 +7,102 @@
<h2 class="directory__title">Сотрудники</h2>
{% set range_start = page.offset + 1 if page.total else 0 %}
{% set range_end = [page.offset + page.employees|length, page.total]|min %}
<p class="directory__summary" role="status">Показаны {{ range_start }}–{{ range_end }} из {{ page.total }}</p>
<p class="directory__summary" role="status" data-directory-summary>Показаны {{ range_start }}–{{ range_end }} из {{ page.total }}</p>
</div>
<div class="directory__actions">
<a class="button button--secondary" href="{{ export_url }}" data-directory-export>Скачать XLSX</a>
<button class="button" type="button" data-columns-open aria-controls="columns-dialog" aria-expanded="false">Настроить колонки</button>
</div>
<button class="button" type="button" data-columns-open aria-controls="columns-dialog" aria-expanded="false">Настроить колонки</button>
</div>
{% set has_filters = filters.q or filters.status or filters.has_email or filters.has_academic_degree or filters.pps or filters.started_from or filters.started_to or filters.has_training or filters.training_since_year %}
<details class="directory__filter-panel">
<summary>Фильтры{% if has_filters %} · применены{% endif %}</summary>
<form class="directory__filters" method="get" action="/admin/directory">
<fieldset class="directory__filter-group directory__filter-group--search">
<legend class="directory__filter-legend">Поиск</legend>
<label class="directory__field">ФИО или ссылка<input class="directory__input" name="q" value="{{ filters.q }}" placeholder="Например, Иванов или hse.ru"></label>
</fieldset>
<fieldset class="directory__filter-group">
<legend class="directory__filter-legend">Статус и данные</legend>
<label class="directory__field">Статус<select class="directory__input" name="status">
<option value="" {% if not filters.status %}selected{% endif %}>Все статусы</option>
<option value="active" {% if filters.status == "active" %}selected{% endif %}>Работает</option>
<option value="verification_required" {% if filters.status == "verification_required" %}selected{% endif %}>Требует проверки</option>
<option value="dismissed" {% if filters.status == "dismissed" %}selected{% endif %}>Уволен</option>
</select></label>
<label class="directory__field">Email<select class="directory__input" name="has_email">
<option value="" {% if not filters.has_email %}selected{% endif %}>Любой email</option>
<option value="true" {% if filters.has_email == "true" %}selected{% endif %}>Есть email</option>
<option value="false" {% if filters.has_email == "false" %}selected{% endif %}>Нет email</option>
</select></label>
<label class="directory__field">Учёная степень<select class="directory__input" name="has_academic_degree">
<option value="" {% if not filters.has_academic_degree %}selected{% endif %}>Любая учёная степень</option>
<option value="true" {% if filters.has_academic_degree == "true" %}selected{% endif %}>Есть учёная степень</option>
<option value="false" {% if filters.has_academic_degree == "false" %}selected{% endif %}>Нет учёной степени</option>
</select></label>
{% set has_extra_filters = filters.status or filters.has_email or filters.has_academic_degree or filters.started_from or filters.started_to or filters.has_training or filters.training_since_year %}
{% set has_filters = filters.q or filters.pps or has_extra_filters %}
<form method="get" action="/admin/directory" data-directory-filters>
<div class="directory__toolbar">
<label class="directory__field directory__search">ФИО или ссылка<input class="directory__input" name="q" value="{{ filters.q }}" placeholder="Введите не менее 4 символов"></label>
<label class="directory__checkbox"><input type="checkbox" name="pps" value="true" {% if filters.pps %}checked{% endif %}> ППС</label>
</fieldset>
<fieldset class="directory__filter-group">
<legend class="directory__filter-legend">Повышение квалификации</legend>
<label class="directory__field">Курсы в профиле<select class="directory__input" name="has_training">
<option value="" {% if not filters.has_training %}selected{% endif %}>Любое</option>
<option value="true" {% if filters.has_training == "true" %}selected{% endif %}>Есть курсы</option>
<option value="false" {% if filters.has_training == "false" %}selected{% endif %}>Нет курсов</option>
</select></label>
<label class="directory__field">С какого года включительно
<input class="directory__input" name="training_since_year" inputmode="numeric" pattern="[1-9][0-9]{3}" maxlength="4" value="{{ filters.training_since_year }}" placeholder="Например, 2024" aria-describedby="training-year-hint">
</label>
<small class="directory__filter-hint" id="training-year-hint">Если указан только год, будут показаны сотрудники с курсами.</small>
</fieldset>
<fieldset class="directory__filter-group">
<legend class="directory__filter-legend">Период обнаружения</legend>
<label class="directory__field">От<input class="directory__input" type="date" name="started_from" value="{{ filters.started_from }}"></label>
<label class="directory__field">До<input class="directory__input" type="date" name="started_to" value="{{ filters.started_to }}"></label>
</fieldset>
<fieldset class="directory__filter-group">
<legend class="directory__filter-legend">Сортировка и страница</legend>
<label class="directory__field">Сортировать по<select class="directory__input" name="sort">
{% for value, label in [("full_name", "ФИО"), ("status", "Статус"), ("hse_start_year", "Год начала"), ("first_seen_at", "Впервые найден"), ("last_seen_at", "Последний раз найден"), ("dismissed_at", "Дата увольнения")] %}
<option value="{{ value }}" {% if filters.sort == value %}selected{% endif %}>{{ label }}</option>
{% endfor %}
</select></label>
<label class="directory__field">Направление<select class="directory__input" name="direction">
<option value="asc" {% if filters.direction == "asc" %}selected{% endif %}>По возрастанию</option>
<option value="desc" {% if filters.direction == "desc" %}selected{% endif %}>По убыванию</option>
<label class="directory__field directory__sort">Сортировать по<select class="directory__input" name="sort">
{% for value, label in [("full_name", "ФИО"), ("status", "Статус"), ("hse_start_year", "Год начала"), ("first_seen_at", "Впервые найден"), ("last_seen_at", "Последний раз найден"), ("dismissed_at", "Дата увольнения")] %}
<option value="{{ value }}" {% if filters.sort == value %}selected{% endif %}>{{ label }}</option>
{% endfor %}
</select></label>
<fieldset class="directory__direction">
<legend>Направление</legend>
{% for value, label in [("asc", "По возрастанию"), ("desc", "По убыванию")] %}
<label title="{{ label }}">
<input type="radio" name="direction" value="{{ value }}" aria-label="{{ label }}" {% if filters.direction == value %}checked{% endif %}>
<span>
<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true">
<path d="{{ 'M12 19V5m-6 6 6-6 6 6' if value == 'asc' else 'M12 5v14m-6-6 6 6 6-6' }}"/>
</svg>
</span>
</label>
{% endfor %}
</fieldset>
<label class="directory__field">Сотрудников на странице<select class="directory__input" name="limit">
{% for value in [25, 50, 100] %}
<option value="{{ value }}" {% if filters.limit == value %}selected{% endif %}>{{ value }}</option>
{% endfor %}
{% for value in [25, 50, 100] %}
<option value="{{ value }}" {% if filters.limit == value %}selected{% endif %}>{{ value }}</option>
{% endfor %}
</select></label>
</fieldset>
<div class="directory__filter-actions">
<button class="button" type="submit">Применить фильтры</button>
<a class="button button--ghost" href="/admin/directory">Сбросить</a>
</div>
<details class="directory__filter-panel">
<summary>Дополнительные фильтры{% if has_extra_filters %} · применены{% endif %}</summary>
<div class="directory__filters">
<fieldset class="directory__filter-group" aria-label="Статус и данные">
<label class="directory__field">Статус<select class="directory__input" name="status">
<option value="" {% if not filters.status %}selected{% endif %}>Все статусы</option>
<option value="active" {% if filters.status == "active" %}selected{% endif %}>Работает</option>
<option value="verification_required" {% if filters.status == "verification_required" %}selected{% endif %}>Требует проверки</option>
<option value="dismissed" {% if filters.status == "dismissed" %}selected{% endif %}>Уволен</option>
</select></label>
<label class="directory__field">Email<select class="directory__input" name="has_email">
<option value="" {% if not filters.has_email %}selected{% endif %}>Любой email</option>
<option value="true" {% if filters.has_email == "true" %}selected{% endif %}>Есть email</option>
<option value="false" {% if filters.has_email == "false" %}selected{% endif %}>Нет email</option>
</select></label>
<label class="directory__field">Учёная степень<select class="directory__input" name="has_academic_degree">
<option value="" {% if not filters.has_academic_degree %}selected{% endif %}>Любая учёная степень</option>
<option value="true" {% if filters.has_academic_degree == "true" %}selected{% endif %}>Есть учёная степень</option>
<option value="false" {% if filters.has_academic_degree == "false" %}selected{% endif %}>Нет учёной степени</option>
</select></label>
</fieldset>
<fieldset class="directory__filter-group" aria-label="Повышение квалификации">
<label class="directory__field">Курсы в профиле<select class="directory__input" name="has_training">
<option value="" {% if not filters.has_training %}selected{% endif %}>Любое</option>
<option value="true" {% if filters.has_training == "true" %}selected{% endif %}>Есть курсы</option>
<option value="false" {% if filters.has_training == "false" %}selected{% endif %}>Нет курсов</option>
</select></label>
<label class="directory__field">С какого года включительно
<input class="directory__input" name="training_since_year" inputmode="numeric" pattern="[1-9][0-9]{3}" maxlength="4" value="{{ filters.training_since_year }}" placeholder="Например, 2024" aria-describedby="training-year-hint">
</label>
<small class="directory__filter-hint" id="training-year-hint">Если указан только год, будут показаны сотрудники с курсами.</small>
</fieldset>
<fieldset class="directory__filter-group" aria-label="Период обнаружения">
<label class="directory__field">От<input class="directory__input" type="date" name="started_from" value="{{ filters.started_from }}"></label>
<label class="directory__field">До<input class="directory__input" type="date" name="started_to" value="{{ filters.started_to }}"></label>
</fieldset>
<div class="directory__filter-actions">
<button class="button" type="submit">Применить фильтры</button>
</div>
</div>
</details>
</form>
</details>
<details class="directory__help">
<summary>Что означают даты и статусы</summary>
<p>«Год начала» — начало работы в ВШЭ. «Впервые найден» и «Последний раз найден» — даты обнаружения сотрудника в списке источника, а не даты приёма и увольнения. «Требует проверки» означает, что профиль недоступен и увольнение ещё не подтверждено.</p>
</details>
<div class="directory__table-wrap">
<div class="directory__results">
<div class="directory__loading" data-directory-loading hidden aria-hidden="true">Обновляем список…</div>
<div class="directory__table-wrap" data-directory-results>
<table class="directory-table" data-directory-table>
<thead>
<tr>
<th class="directory-table__head" data-column="full_name">ФИО</th>
<th class="directory-table__head" data-column="status">Статус</th>
<th class="directory-table__head" data-column="positions">Должности</th>
<th class="directory-table__head" data-column="hse_start_year">Год начала</th>
<th class="directory-table__head" data-column="email">Email</th>
<th class="directory-table__head" data-column="phone">Телефон</th>
<th class="directory-table__head" data-column="address">Адрес</th>
<th class="directory-table__head" data-column="academic_degree">Учёная степень</th>
<th class="directory-table__head" data-column="publications_count">Публикации</th>
<th class="directory-table__head" data-column="courses_count">Курсы</th>
<th class="directory-table__head" data-column="news_count">Новости</th>
<th class="directory-table__head" data-column="first_seen_at">Впервые найден</th>
<th class="directory-table__head" data-column="last_seen_at">Последний раз найден</th>
<th class="directory-table__head" data-column="dismissed_at">Дата увольнения</th>
<th class="directory-table__head" data-column="profile">Профиль</th>
{% for key, label, _ in directory_columns %}
<th class="directory-table__head" data-column="{{ key }}">{{ label }}</th>
{% endfor %}
</tr>
</thead>
<tbody>
@@ -125,9 +123,10 @@
<td class="directory-table__cell" data-column="last_seen_at">{{ employee.last_seen_display }}</td>
<td class="directory-table__cell" data-column="dismissed_at">{{ employee.dismissed_display }}</td>
<td class="directory-table__cell" data-column="profile"><a class="admin__link" href="{{ employee.canonical_url }}" target="_blank" rel="noopener">Внешний профиль</a></td>
<td class="directory-table__cell" data-column="dpo_count">{{ employee.dpo_count if employee.dpo_count is not none else "—" }}{% if employee.dpo_uncertain %} <a class="dpo-warning" href="/admin/employees/{{ employee.id }}#unresolved-dpo" title="Часть сведений о ДПО не распознана. Количество может быть неполным." aria-label="Часть сведений о ДПО не распознана. Количество может быть неполным."><span aria-hidden="true">⚠︎</span></a>{% endif %}</td>
</tr>
{% else %}
<tr><td class="directory-table__empty" colspan="15">
<tr><td class="directory-table__empty" colspan="{{ directory_columns|length }}">
<strong>Сотрудники не найдены</strong>
{% if has_filters %}
<span>Попробуйте изменить запрос или сбросить фильтры.</span>
@@ -140,8 +139,9 @@
</tbody>
</table>
</div>
</div>
<div class="directory__pagination">
<div class="directory__pagination" data-directory-pagination>
{% set prev_offset = filters.offset - filters.limit %}
{% set next_offset = filters.offset + filters.limit %}
{% if filters.offset > 0 %}
@@ -161,7 +161,7 @@
<button class="button button--ghost" type="button" data-columns-close>Закрыть</button>
</div>
<div class="columns-modal__grid">
{% for key, label in [("full_name", "ФИО"), ("status", "Статус"), ("positions", "Должности"), ("hse_start_year", "Год начала"), ("email", "Email"), ("phone", "Телефон"), ("address", "Адрес"), ("academic_degree", "Учёная степень"), ("publications_count", "Публикации"), ("courses_count", "Курсы"), ("news_count", "Новости"), ("first_seen_at", "Впервые найден"), ("last_seen_at", "Последний раз найден"), ("dismissed_at", "Дата увольнения"), ("profile", "Профиль")] %}
{% for key, label, _ in directory_columns %}
<label class="columns-modal__option"><input class="columns-modal__checkbox" type="checkbox" value="{{ key }}" data-column-toggle> {{ label }}</label>
{% endfor %}
</div>
@@ -175,5 +175,5 @@
</dialog>
{% endblock %}
{% block scripts %}
<script src="/static/admin.js"></script>
<script src="/static/admin.js?v={{ frontend_version }}"></script>
{% endblock %}
+12 -1
View File
@@ -4,7 +4,7 @@
<section class="employee-card">
<div class="employee-card__header">
<div class="employee-card__identity">
<h2 class="employee-card__title">{{ employee_view.full_name or employee.profile_key }}</h2>
<h2 class="employee-card__title">{{ employee_view.full_name or employee.profile_key }}{% if employee_view.dpo_uncertain %} <a class="dpo-warning" href="#unresolved-dpo" title="Часть сведений о ДПО не распознана. Количество может быть неполным." aria-label="Часть сведений о ДПО не распознана. Количество может быть неполным."><span aria-hidden="true">⚠︎</span></a>{% endif %}</h2>
<span class="badge {% if employee_view.status == "dismissed" %}badge--dismissed{% endif %}">{{ employee_view.status_display }}</span>
</div>
<div class="employee-card__actions">
@@ -215,6 +215,9 @@
</table>
</div>
{% else %}
{% if section.type == "additional_education" and section.list_items %}
<p class="employee-section__note">Всего: {{ section.list_items | length }}</p>
{% endif %}
{% if section.paragraphs %}
{% for paragraph in section.paragraphs %}
<p class="employee-section__text">{{ paragraph }}</p>
@@ -238,6 +241,14 @@
{% endif %}
</article>
{% endfor %}
{% if employee_view.dpo_uncertain %}
<div id="unresolved-dpo" tabindex="-1">
{% for section in profile_sections %}
{% for text in section.unresolved %}<p>{{ text }}</p>{% endfor %}
{% for doc in section.unresolved_documents %}<p>{% if doc.url.startswith(('https://', 'http://')) %}<a href="{{ doc.url }}">{{ doc.text }}</a>{% else %}{{ doc.text }}{% endif %}</p>{% endfor %}
{% endfor %}
</div>
{% endif %}
</div>
{% else %}
<p class="employee-section__text">Разделы профиля не найдены.</p>
+3 -2
View File
@@ -13,13 +13,14 @@
</section>
<section class="guide__step" data-guide-step>
<h3 class="guide__heading" tabindex="-1">Найдите сотрудника и откройте профиль</h3>
<p>В разделе «Сотрудники» раскройте панель «Фильтры». В поле «ФИО или ссылка» введите фамилию, часть ФИО или ссылку на профиль ВШЭ. Например, Иванов.</p>
<p>Введите фамилию, часть ФИО или ссылку на профиль ВШЭ. Поиск срабатывает с четвёртого символа. Отметьте «ППС», чтобы оставить профессорско-преподавательский состав. Остальные условия находятся в панели «Дополнительные фильтры» и применяются её кнопкой.</p>
<dl class="guide__fields">
<dt>Статус, Email, Учёная степень</dt><dd>Ограничивают список по состоянию сотрудника и наличию сведений в профиле. Звание профессора или доцента не заменяет учёную степень.</dd>
<dt>Повышение квалификации</dt><dd>Выберите «Есть курсы» или «Нет курсов» по записям в профиле ВШЭ. Например, «Нет курсов» и 2024 покажут сотрудников без записей о курсах за 2024 год и позже. Если указать только год, отобразятся сотрудники с курсами за этот период.</dd>
<dt>Период обнаружения: От / До</dt><dd>Введите даты первого обнаружения профиля системой. Это не период приёма на работу.</dd>
<dt>Сортировка и страница</dt><dd>Выберите порядок и число строк. «Применить фильтры» обновит список, «Сбросить» вернёт исходные параметры.</dd>
<dt>Порядок списка</dt><dd>Поле сортировки, направление и число строк сразу обновляют список. «Сбросить» вернёт исходные параметры.</dd>
<dt>Настроить колонки</dt><dd>Позволяет выбрать видимые сведения: должности, контакты, степень, число публикаций, курсов и новостей, даты обнаружения.</dd>
<dt>Скачать XLSX</dt><dd>В файл попадут все колонки и все сотрудники, найденные с применёнными фильтрами, в выбранном порядке.</dd>
</dl>
<p>Откройте ФИО в таблице: в карточке собраны должности, статус, контакты, внешние идентификаторы, новости и доступные разделы профиля — образование, степени, публикации, курсы и выпускные работы. Состав зависит от данных источника.</p>
<p>«Год начала» — начало работы в ВШЭ. «Впервые найден» и «Последний раз найден» — наблюдения системы. «Требует проверки» означает, что профиль недоступен, а увольнение ещё не подтверждено.</p>
+3 -3
View File
@@ -1,3 +1,3 @@
APP_VERSION = "2.0.1"
FRONTEND_VERSION = "2.0.1"
BACKEND_VERSION = "2.0.1"
APP_VERSION = "2.3.0"
FRONTEND_VERSION = "2.3.0"
BACKEND_VERSION = "2.3.0"
+1 -1
View File
@@ -1,6 +1,6 @@
[project]
name = "miem-workers"
version = "2.0.1"
version = "2.3.0"
description = "MIEM employees parser, admin API, and web admin"
requires-python = ">=3.11"
dependencies = [
+1 -1
View File
@@ -77,7 +77,7 @@ def main():
filters = page.locator(".directory__filter-panel")
expect(filters).not_to_have_attribute("open", "")
expect(page.get_by_label("ФИО или ссылка")).to_be_hidden()
expect(page.get_by_label("ФИО или ссылка")).to_be_visible()
filters.locator("summary").focus()
page.keyboard.press("Enter")
expect(page.get_by_label("ФИО или ссылка")).to_be_visible()
+176
View File
@@ -0,0 +1,176 @@
"""Directory request states with delayed responses; no production access.
Run: python tests/browser_directory_loading.py (Playwright + Chromium required).
"""
import os
from pathlib import Path
import sys
from tempfile import mkdtemp
from urllib.parse import urlsplit
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
from fastapi.testclient import TestClient
from playwright.sync_api import expect, sync_playwright
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy.pool import StaticPool
from app.config import Settings, get_settings
from app.db import Base, get_db
from app.main import app
from app.models import Employee
from app.security import SESSION_COOKIE, sign_session
def main():
engine = create_engine("sqlite://", connect_args={"check_same_thread": False}, poolclass=StaticPool)
Base.metadata.create_all(engine)
factory = sessionmaker(bind=engine)
with factory() as db:
db.add_all([
Employee(profile_key="staff:1", canonical_url="https://www.hse.ru/staff/1",
full_name="Иванов Иван", current_data={"positions": ["Кафедра математики"]}),
Employee(profile_key="staff:2", canonical_url="https://www.hse.ru/staff/2",
full_name="Петров Пётр", current_data={"positions": ["Администрация"]}),
])
db.commit()
def database():
with factory() as db:
yield db
settings = Settings(_env_file=None, session_secret="browser-loading-test-secret")
app.dependency_overrides[get_db] = database
app.dependency_overrides[get_settings] = lambda: settings
client = TestClient(app)
client.cookies.set(SESSION_COOKIE, sign_session(settings.admin_username, settings))
pending = []
output = Path(mkdtemp(prefix="miem-directory-loading-"))
def fulfill(route):
url = urlsplit(route.request.url)
response = client.get(url.path + ("?" + url.query if url.query else ""))
route.fulfill(status=response.status_code, body=response.content,
content_type=response.headers.get("content-type", "text/plain"))
def serve(route):
if route.request.resource_type == "fetch" and urlsplit(route.request.url).path == "/admin/directory":
pending.append(route)
else:
fulfill(route)
try:
with sync_playwright() as playwright:
browser = playwright.chromium.launch(executable_path=os.getenv("BROWSER_EXECUTABLE"))
page = browser.new_page(viewport={"width": 1440, "height": 1000})
page.route("**/*", serve)
errors = []
page.on("pageerror", lambda error: errors.append(str(error)))
page.goto("http://miem.test/admin/directory")
summary = page.locator("[data-directory-summary]")
results = page.locator("[data-directory-results]")
overlay = page.locator("[data-directory-loading]")
search = page.get_by_label("ФИО или ссылка")
pps = page.get_by_label("ППС", exact=True)
apply = page.get_by_role("button", name="Применить фильтры", include_hidden=True)
columns = page.get_by_role("button", name="Настроить колонки")
panel = page.locator(".directory__filter-panel")
# Immediate filter feedback, actual disabled controls, inert links.
pps.check()
expect(overlay).to_be_visible()
expect(summary).to_have_text("Обновляем список…")
expect(pps).to_be_disabled()
expect(columns).to_be_disabled()
expect(apply).to_be_disabled()
expect(search).to_be_disabled()
expect(page.locator("[data-directory-export]")).to_have_attribute("inert", "")
expect(results).to_have_attribute("aria-busy", "true")
page.wait_for_timeout(100)
assert len(pending) == 1
assert "pps=true" in pending[0].request.url
page.screenshot(path=str(output / "desktop-loading.png"), full_page=True)
page.set_viewport_size({"width": 390, "height": 844})
assert page.evaluate("document.documentElement.scrollWidth <= innerWidth")
page.screenshot(path=str(output / "mobile-loading.png"), full_page=True)
fulfill(pending.pop(0))
expect(summary).to_have_text("Показаны 1–1 из 1")
expect(pps).to_be_enabled()
expect(overlay).to_be_hidden()
expect(results).to_have_attribute("aria-busy", "false")
assert "pps=True" in page.locator("[data-directory-export]").get_attribute("href")
# Live search leaves ALL controls usable, including during debounce.
search.fill("Иван")
expect(summary).to_have_text("Идёт поиск…")
expect(search).to_be_enabled()
expect(pps).to_be_enabled()
expect(columns).to_be_enabled()
expect(overlay).to_be_hidden()
page.wait_for_timeout(350)
assert len(pending) == 1
stale = pending.pop(0)
search.fill("Петр")
page.wait_for_timeout(350)
assert len(pending) == 1
fulfill(pending.pop(0))
expect(summary).to_have_text("Показаны 0–0 из 0")
fulfill(stale)
page.wait_for_timeout(100)
expect(summary).to_have_text("Показаны 0–0 из 0")
expect(search).to_have_value("Петр")
# Cancelling back to the applied query must clear the indicator.
search.fill("Петров")
page.wait_for_timeout(350)
stale = pending.pop(0)
search.fill("Петр")
expect(summary).to_have_text("Показаны 0–0 из 0")
fulfill(stale)
page.wait_for_timeout(350)
assert not pending
expect(results).to_have_attribute("aria-busy", "false")
# Manual submission serializes values BEFORE disabling controls.
panel.locator("summary").click()
panel.locator('[name="has_email"]').select_option("true")
apply.click()
expect(apply).to_be_disabled()
expect(overlay).to_be_visible()
page.wait_for_timeout(100)
assert len(pending) == 1
assert "has_email=true" in pending[0].request.url and "pps=true" in pending[0].request.url
fulfill(pending.pop(0))
expect(overlay).to_be_hidden()
expect(panel.locator("summary")).to_contain_text("применены")
assert "has_email=true" in page.url
# Failure preserves results and releases every lock; retry works.
pps.uncheck()
expect(overlay).to_be_visible()
page.wait_for_timeout(100)
pending.pop(0).fulfill(status=503, body="unavailable")
expect(summary).to_contain_text("Не удалось обновить список")
expect(overlay).to_be_hidden()
expect(pps).to_be_enabled()
expect(columns).to_be_enabled()
expect(search).to_be_enabled()
expect(page.locator("[data-directory-export]")).not_to_have_attribute("inert", "")
panel.locator("summary").click()
apply.click()
page.wait_for_timeout(100)
fulfill(pending.pop(0))
expect(summary).to_have_text("Показаны 0–0 из 0")
expect(overlay).to_be_hidden()
assert not errors, errors
browser.close()
finally:
app.dependency_overrides.clear()
client.close()
engine.dispose()
print(f"Directory loading checks passed; screenshots: {output}")
if __name__ == "__main__":
main()
-1
View File
@@ -29,7 +29,6 @@ def test_directory_template_is_russian_and_uses_display_dates():
assert "Сотрудников на странице" in template
assert "{% for value in [25, 50, 100] %}" in template
assert "Показаны" in template
assert "Новости" in template
assert "Есть учёная степень" in template
assert 'data-column="academic_degree"' in template
assert "employee.news_count" in template
+63 -1
View File
@@ -1,7 +1,11 @@
from datetime import datetime, timezone
from io import BytesIO
from types import SimpleNamespace
from urllib.parse import parse_qs, urlsplit
from bs4 import BeautifulSoup
from fastapi.testclient import TestClient
from openpyxl import load_workbook
from sqlalchemy import create_engine, select
from sqlalchemy.exc import OperationalError
from sqlalchemy.orm import sessionmaker
@@ -18,7 +22,7 @@ def test_health_returns_versions():
response = TestClient(app).get("/api/health")
assert response.status_code == 200
assert response.json()["backend_version"] == "2.0.1"
assert response.json()["backend_version"] == "2.3.0"
def test_api_employees_and_stats_require_admin_session():
@@ -153,3 +157,61 @@ def test_directory_database_error_has_retry_and_preserves_filters(db_session, mo
assert "private database details" not in response.text
finally:
app.dependency_overrides.clear()
def test_directory_export_uses_applied_filters_and_sort_across_pages(db_session):
for number in range(30):
db_session.add(Employee(
profile_key=f"export:{number}",
canonical_url=f"https://www.hse.ru/staff/export-{number}",
full_name=f"Export {number:02d}",
status="dismissed" if number == 28 else "active",
current_data={
"hse_start_year": 2025 - number,
"positions": ["Инженер" if number == 29 else "Кафедра физики"],
"contacts": {"address": "=1+1", "phones": ["+7 123"]},
},
))
db_session.commit()
settings = Settings(_env_file=None, session_secret="test-session-secret")
app.dependency_overrides[get_db] = lambda: db_session
app.dependency_overrides[get_settings] = lambda: settings
try:
client = TestClient(app)
anonymous = client.get("/admin/directory/export.xlsx", follow_redirects=False)
assert anonymous.status_code == 303 and anonymous.headers["location"] == "/admin/login"
client.cookies.set(SESSION_COOKIE, sign_session(settings.admin_username, settings))
response = client.get("/admin/directory", params={
"q": "Export", "pps": "true", "status": "active", "sort": "hse_start_year",
"direction": "desc", "limit": 25, "offset": 25,
})
assert response.status_code == 200
page = BeautifulSoup(response.text, "html.parser")
assert "Показаны 26–28 из 28" in page.get_text()
export_url = page.find("a", string="Скачать XLSX")["href"]
assert parse_qs(urlsplit(export_url).query) == {
"q": ["Export"], "pps": ["True"], "status": ["active"], "sort": ["hse_start_year"], "direction": ["desc"],
}
export = client.get(export_url + "&limit=25&offset=25")
assert export.status_code == 200
assert export.headers["content-disposition"] == 'attachment; filename="employees.xlsx"'
assert export.headers["cache-control"] == "no-store"
sheet = load_workbook(BytesIO(export.content)).active
assert [cell.value for cell in sheet[1]] == [
cell.get_text(strip=True) for cell in page.select(".directory-table th")
]
assert sheet.max_column == 16
assert [cell.value for cell in sheet["A"][1:]] == [f"Export {number:02d}" for number in range(28)]
assert [cell.value for cell in sheet["A"][-3:]] == [
cell.get_text(strip=True) for cell in page.select("tbody [data-column='full_name']")
]
assert sheet["F2"].value == "+7 123"
assert sheet["G2"].value == "=1+1" and sheet["G2"].data_type == "s"
assert sheet["O2"].value == "https://www.hse.ru/staff/export-0"
empty = client.get("/admin/directory/export.xlsx?q=missing")
assert empty.status_code == 200
assert load_workbook(BytesIO(empty.content)).active.max_row == 1
finally:
app.dependency_overrides.clear()
+1 -1
View File
@@ -230,7 +230,7 @@ def test_public_page_data_minimization_and_safe_links(db_session, export):
assert "javascript:" not in fragment.text and "private@example.org" not in fragment.text
assert "javascript:" not in response.text and "private@example.org" not in response.text
assert "/admin/employees/" not in response.text and "/admin/logout" not in response.text
assert "Backend 2.0.1 · Frontend 2.0.1" in response.text
assert "Backend 2.3.0 · Frontend 2.3.0" in response.text
assert "Вход в систему" in response.text
assert client.get("/admin/directory", follow_redirects=False).status_code in (302, 303, 401, 403)
assert "Сотрудники не найдены" in client.get("/admin/council?q=missing").text
+29
View File
@@ -0,0 +1,29 @@
from bs4 import BeautifulSoup
from app.parser.education_document import _certificate
from app.parser.profile import extract_sections
from app.services.resource_cache import ResourceCache
def test_education_events_preserve_ambiguities_and_separate_periods():
row = '<div><div class="person-list-hangover">{}</div><span>«Первая помощь»</span>, повышение квалификации, {}</div>'
html = ('<h2>Дополнительное образование</h2>' + row.format(2025, "НИУ ВШЭ") +
row.format(2025, "Высшая школа экономики") + row.format(2023, "НИУ ВШЭ") +
'<p>Прошел тест с присвоением степени</p><p>«Инженер»</p>'
'<p>Дополнительные сведения</p><p><a href="/mirror/pubs/share/1">Документ</a></p>')
section, = extract_sections(BeautifulSoup(html, "html.parser"), "https://www.hse.ru", lambda url: None)
assert len(section["items"]) == 3
assert section["items"][-1] == "Прошел тест с присвоением степени «Инженер»"
assert section["unresolved"] == ["Дополнительные сведения"]
assert section["unresolved_documents"][0]["url"] == "https://www.hse.ru/mirror/pubs/share/1"
def test_document_read_and_cache(db_session):
text = 'НИУ ВШЭ. Удостоверение о повышении квалификации по программе «Деловой английский» в объеме 24 часов. 2025 год'
result = _certificate(text)
assert result == "2025 «Деловой английский», повышение квалификации, НИУ ВШЭ, 24 часов"
assert _certificate("Сертификат 2025") is None
cache = ResourceCache(db_session)
assert cache.education_document("staff:test", "https://www.hse.ru/test", "1" * 64, lambda: result) == result
assert cache.education_document("staff:test", "https://www.hse.ru/test", "1" * 64, lambda: 1 / 0) == result
+31
View File
@@ -4,6 +4,37 @@ from app.parser.profile import enrich_sections_from_hse_widgets, extract_person_
from app.parser.profile_url import normalize_profile_url, parse_profile_identity
def test_mixed_additional_education_keeps_all_courses_and_omits_attachments():
structured = "".join(
f'<div class="g-ul"><div class="with-indent"><div class="person-list-hangover">{year}</div>'
f'<span>{title}</span><span>, {provider}</span></div></div>'
for year, title, provider in [(2025, "Экспортный контроль", "МИФИ"),
(2024, "Smart LMS", "ВШЭ"), (2018, "Методическая работа", "Дирекция")]
)
html = f'''<div tab-node="additional_education"><h2 class="c_14">Дополнительное образование</h2>
{structured}<div class="with-indent"><ul><li>2020 г. НИУ ВШЭ, организация учебного процесса</li></ul>
<p><a href="/f.pdf">Файл</a> (PDF, 407 Кб)</p>
<ul><li>2017 г. МГУ, методика <a href="/mirror/pubs/share/123">Сертификат</a></li>
<li>2016 г. КФУ, образовательные стандарты</li><li>2015 г. ЮФУ, ФГОС</li>
<li><strong>2015 г</strong>. МГТУ МИРЭА.</li></ul>
<p class="text">Направления совершенствования практической подготовки.</p>
<p><a href="/cert.jpg">Сертификат_МИРЭА</a> (JPG, 7,28 Мб).</p>
<ul><li>2014 г. АИОР, аккредитация.</li></ul>
<p><a href="/certificate">Свидетельство</a> (PDF, 9,36 Мб).</p>
<ul><li>2011 г. РАНХиГС.</li></ul>
<p>Курс <a href="/programme">Современные подходы</a>.</p>
<p><a href="/cert.jpg">Удостоверение</a> (JPG, 6,69 Мб).</p></div></div>'''
section, = extract_sections(BeautifulSoup(html, "html.parser"), "https://www.hse.ru/")
assert len(section["items"]) == 10
assert [row["year"] for row in section["year_entries"]] == [2025, 2024, 2018]
assert section["items"][0] == "2025 Экспортный контроль, МИФИ"
assert section["items"][7] == "2015 г. МГТУ МИРЭА. Направления совершенствования практической подготовки."
assert section["items"][9].startswith("2011 г. РАНХиГС. Курс Современные подходы")
assert section["links"] == [{"text": "Современные подходы", "url": "https://www.hse.ru/programme"}]
for fragment in ("Сертификат", "Файл", "Свидетельство", "Удостоверение", "PDF", "JPG"):
assert fragment not in str(section)
class FakeResponse:
def __init__(self, payload):
self.payload = payload
+1 -1
View File
@@ -99,7 +99,7 @@ def test_training_filter_uses_course_year_and_kind_before_pagination(db_session)
page = BeautifulSoup(response.text, "html.parser")
panel = page.select_one(".directory__filter-panel")
assert not panel.has_attr("open")
assert panel.summary.get_text(strip=True) == "Фильтры · применены"
assert panel.summary.get_text(strip=True) == "Дополнительные фильтры · применены"
assert panel.select_one('select[name="has_training"] option[selected]')["value"] == "true"
assert panel.select_one('input[name="training_since_year"]')["value"] == "2024"
assert "Person 1" not in response.text