Добавить реестр доступности учебной литературы ВШЭ #1
No files matched your search
@@ -0,0 +1,22 @@
|
||||
.git
|
||||
node_modules
|
||||
.next
|
||||
.wrangler
|
||||
.sites-runtime
|
||||
dist
|
||||
work
|
||||
outputs
|
||||
data
|
||||
.env*
|
||||
*.sqlite*
|
||||
*.tsbuildinfo
|
||||
components
|
||||
hooks
|
||||
examples
|
||||
db
|
||||
drizzle
|
||||
scripts
|
||||
worker
|
||||
vendor
|
||||
.openai
|
||||
app/chatgpt-auth.ts
|
||||
+41
@@ -0,0 +1,41 @@
|
||||
# Next.js / Node
|
||||
node_modules/
|
||||
.next/
|
||||
next-env.d.ts
|
||||
*.tsbuildinfo
|
||||
# Persistent runtime data and local state
|
||||
data/
|
||||
*.sqlite*
|
||||
.env*
|
||||
.wrangler/
|
||||
.sites-runtime/
|
||||
.vinext/
|
||||
dist/
|
||||
work/
|
||||
outputs/
|
||||
# Retained original Sites scaffold (not part of Node deployment)
|
||||
components/
|
||||
hooks/
|
||||
examples/
|
||||
db/
|
||||
drizzle/
|
||||
scripts/
|
||||
worker/
|
||||
vendor/
|
||||
.openai/
|
||||
vite.config.ts
|
||||
drizzle.config.ts
|
||||
cloudflare-env.d.ts
|
||||
components.json
|
||||
eslint.config.mjs
|
||||
lib/utils.ts
|
||||
lib/connectors.ts
|
||||
lib/connector-preview.d.ts
|
||||
lib/connector-errors.mts
|
||||
lib/connector-contract.mts
|
||||
lib/connector-context.ts
|
||||
app/chatgpt-auth.ts
|
||||
# OS
|
||||
.DS_Store
|
||||
AGENTS.md
|
||||
CLAUDE.md
|
||||
@@ -0,0 +1,52 @@
|
||||
# Changelog
|
||||
|
||||
## 1.3.0 — 2026-10-30
|
||||
|
||||
- Проверка литературы учитывает доступ к полному тексту без авторизации: вход, подписка, покупка онлайн-доступа и ознакомительный режим отмечаются как причины недоступности.
|
||||
- Добавлены правила для Лань, Юрайт, Znanium, iBooks, Springer и BOOK.ru с браузерной проверкой видимого содержимого и ограниченными переходами в читатель.
|
||||
- Причина и подтверждающая цитата доступны рядом с результатом; HTTP 200, аннотация и PDF сами по себе больше не подтверждают доступность полного текста.
|
||||
- Старые результаты и истёкшие проверки перепроверяются независимо от изменения курса; версия правил и срок кэша предотвращают повторное использование прежних HTTP-оценок.
|
||||
- Chromium работает в изолированном контейнере с проверкой публичных адресов, ограничениями ресурсов и сохранением двухсекундной паузы между переходами к одному хосту.
|
||||
- Увеличен таймаут только маршрута /courses для браузерной проверки; причины недоступности адаптированы для мобильных экранов.
|
||||
|
||||
## 1.2.2 — 2026-10-29
|
||||
|
||||
- Интервал между запросами к одному хосту установлен в две секунды вместо случайной паузы 3–4 секунды.
|
||||
|
||||
## 1.2.1 — 2026-10-29
|
||||
|
||||
- Материалы проверяются последовательно; запросы к одному хосту разделены паузой 3–4 секунды, включая перенаправления.
|
||||
- Паузы и ограничения общие для ручных и автоматических проверок и сохраняются в PostgreSQL после перезапуска.
|
||||
- HTTP429/503 учитывают срок Retry-After с минимальной паузой в минуту; HTTP403 приостанавливает обращения на пять минут.
|
||||
- Пропущенные из-за ограничения ссылки показывают причину и срок паузы; повторные запросы в цикле не выполняются.
|
||||
- Общий таймаут загрузки увеличен до 30 секунд с учётом пауз на перенаправлениях.
|
||||
|
||||
## 1.2.0 — 2026-10-29
|
||||
|
||||
- Хранилище перенесено на PostgreSQL для использования существующего сервера VPS; приложению предназначены отдельная база и роль.
|
||||
- Добавлено общее редактируемое расписание: ежедневная или еженедельная проверка по московскому времени, выбор режима кэша, включение и отключение.
|
||||
- Проверки по расписанию выполняются на сервере при закрытом браузере и возобновляются после перезапуска.
|
||||
- Расписание выключено по умолчанию; изменение или отключение не прерывает уже начатую автоматическую проверку.
|
||||
- Следующий запуск, пропуск совпавшего запуска и ошибки фоновой проверки отображаются в интерфейсе.
|
||||
- Расписание и запуск сохраняются атомарно; совпавшие запуски не дублируются, после простоя выполняется один пропущенный запуск.
|
||||
- При старте приложения одновременно запускаются веб-сервер и worker с общим завершением при сбое одного процесса.
|
||||
|
||||
## 1.1.1 — 2026-10-29
|
||||
|
||||
- Ответы 302/304 без материала, включая бинарный Content-Type, отмечаются как требующие ручной проверки.
|
||||
|
||||
## 1.1.0 — 2026-10-29
|
||||
|
||||
- Перенесён сервер с Cloudflare Workers/D1 на Node.js24 и постоянную SQLite для VPS.
|
||||
- Добавлен путь `/courses` для интерфейса, API и шрифтов; production Origin настроен для miem.space.
|
||||
- DNS-адрес закрепляется в сетевом соединении, включая повторную проверку перенаправлений.
|
||||
- Добавлены standalone-сборка, непривилегированный Docker runtime, постоянный volume, healthcheck и Nginx-сниппет.
|
||||
- Сохранение порций и очереди атомарно; добавлены проверки восстановления после ошибки записи, блокировки одновременных порций и Origin за reverse proxy.
|
||||
|
||||
## 1.0.1 — 2026-10-29
|
||||
|
||||
- Добавлен общий реестр доступности литературы пяти программ НИУ ВШЭ без регистрации.
|
||||
- Ручной запуск с продолжением, обход пагинации, чтение основной и дополнительной литературы, включая текстовые URL.
|
||||
- Сохранение результатов и контрольных сумм курсов; повторное использование неизменившихся курсов и полная проверка без кэша.
|
||||
- Отдельные причины недоступности, переходы к материалам и исходным курсам, поиск, фильтры и экспорт CSV.
|
||||
- Явные отметки неполного охвата, JavaScript-заглушек, ограничений доступа и устаревших результатов.
|
||||
@@ -0,0 +1,2 @@
|
||||
# Визуальная система
|
||||
Режим Operate. Академический реестр: тёплая бумага, глубокий зелёный, киноварь для проблем. Боковая навигация по программам, заголовок с засечками, плотный список результатов. Состояния подписаны словами. Без иллюстративных статистик и декоративных карточек. Адаптивность, клавиатура, заметный фокус, reduced motion.
|
||||
+29
@@ -0,0 +1,29 @@
|
||||
FROM node:24-bookworm-slim@sha256:c86ecead9e55a5210ac1727de38a4ca840bf03dcaf34dc9749addbf383b1920c AS build
|
||||
WORKDIR /app
|
||||
ENV NEXT_TELEMETRY_DISABLED=1
|
||||
RUN npm install --global pnpm@11.25.0
|
||||
COPY package.json pnpm-lock.yaml pnpm-workspace.yaml ./
|
||||
RUN pnpm install --frozen-lockfile
|
||||
ENV PLAYWRIGHT_BROWSERS_PATH=/opt/chromium
|
||||
RUN node node_modules/playwright/cli.js install chromium --only-shell
|
||||
COPY app ./app
|
||||
COPY lib ./lib
|
||||
COPY public ./public
|
||||
COPY next.config.ts tsconfig.json postcss.config.mjs ./
|
||||
RUN pnpm build
|
||||
|
||||
FROM node:24-bookworm-slim@sha256:c86ecead9e55a5210ac1727de38a4ca840bf03dcaf34dc9749addbf383b1920c AS runtime
|
||||
WORKDIR /app
|
||||
ENV NODE_ENV=production NEXT_TELEMETRY_DISABLED=1 HOSTNAME=0.0.0.0 PORT=3000
|
||||
ENV PLAYWRIGHT_BROWSERS_PATH=/opt/chromium
|
||||
COPY --from=build --chown=node:node /app/.next/standalone ./
|
||||
COPY --from=build --chown=node:node /app/.next/static ./.next/static
|
||||
COPY --from=build --chown=node:node /app/public ./public
|
||||
COPY --from=build --chown=node:node /app/lib ./lib
|
||||
COPY server ./server
|
||||
COPY --from=build /opt/chromium /opt/chromium
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends libasound2 libatk-bridge2.0-0 libatk1.0-0 libatspi2.0-0 libcairo2 libcups2 libdbus-1-3 libdrm2 libgbm1 libglib2.0-0 libnspr4 libnss3 libpango-1.0-0 libx11-6 libxcb1 libxcomposite1 libxdamage1 libxext6 libxfixes3 libxkbcommon0 libxrandr2 fonts-liberation fonts-noto-color-emoji && chmod -R a+rX /opt/chromium && rm -rf /var/lib/apt/lists/*
|
||||
USER node
|
||||
EXPOSE 3000
|
||||
HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 CMD node -e "fetch('http://127.0.0.1:3000/courses/api/health').then(r=>{if(!r.ok)process.exit(1)}).catch(()=>process.exit(1))"
|
||||
CMD ["node","server/runtime.mjs","--standalone"]
|
||||
@@ -0,0 +1,8 @@
|
||||
# Сверка
|
||||
Общий реестр доступности литературы пяти программ НИУ ВШЭ. Без входа. Только реальные результаты; начальное состояние пустое. Node.js24 и PostgreSQL хранят результаты, контрольные суммы курсов и прогресс в постоянном volume. Проверка порциями, пока открыта вкладка; продолжение вручную. Неизменившиеся курсы используют кэш; доступна полная проверка. Успешный HTTP-ответ не доказывает доступ к полному тексту. Целевой путь — https://miem.space/courses.
|
||||
|
||||
Расписание: одно общее, ежедневно или еженедельно по Москве, выключено до сохранения пользователем. Запланированный аудит работает фоновым процессом при закрытом браузере и продолжается после перезапуска. Совпавший слот пропускается, если предыдущая ручная или автоматическая проверка ещё running. Изменение/отключение влияет на будущие запуски.
|
||||
|
||||
Для VPS используется отдельная база/роль на существующем PostgreSQL. Пароль только в окружении, не в исходниках.
|
||||
|
||||
Полный текст: авторизация, подписка, покупка или гостевой фрагмент означают недоступность книги с конкретным доказательством. Аннотация и успешный HTTP не дают зелёный статус. Неопределённость и остановка проверки защитой сайта показываются отдельно. Для известных ЭБС доступ проверяется после JS и перехода в читалку без авторизации. Кэш внешних ссылок имеет собственный TTL и версию правил.
|
||||
@@ -0,0 +1,69 @@
|
||||
# Сверка
|
||||
|
||||
Проверка доступности литературы пяти образовательных программ НИУ ВШЭ. Целевой адрес: `https://miem.space/courses`. Авторизация не требуется.
|
||||
|
||||
## Запуск
|
||||
|
||||
Требуется Node.js24 и pnpm11.25.0.
|
||||
|
||||
```sh
|
||||
pnpm install --frozen-lockfile
|
||||
TEST_DATABASE_URL=<изолированная-локальная-БД-sverka_test> pnpm test
|
||||
pnpm typecheck
|
||||
pnpm dev
|
||||
```
|
||||
|
||||
Открыть `http://127.0.0.1:5173/courses`. Для сервера требуется `DATABASE_URL` с подключением к отдельной базе PostgreSQL. Там хранятся результаты, очередь и расписание; новое пустое хранилище инициализируется при первом подключении. Пароль передаётся только через окружение или защищённый файл окружения вне Git. `COURSES_ORIGIN` задаёт допустимый Origin для POST за reverse proxy, в production — `https://miem.space`.
|
||||
|
||||
Ручная проверка выполняется порциями, пока вкладка открыта. После её закрытия или перезапуска сервера кнопка «Продолжить» использует сохранённую очередь. Глобальная аренда не допускает одновременную обработку одной порции. Каждая порция сохраняет результаты и очередь в одной транзакции PostgreSQL; при ошибке записи задания остаются в очереди.
|
||||
|
||||
Страницы курсов сравниваются по контрольной сумме. Неизменившийся список литературы переиспользуется, но доступность внешних ссылок проверяется отдельно: кэш сутки для подтверждённых ограничений, десять минут для неопределённых результатов и временных ошибок. Версия правил входит в проверку свежести; старые HTTP-only результаты показываются как устаревшие и при следующем запуске пересчитываются. Полная проверка игнорирует кэш. Литература только во вложении отмечается как неполный охват.
|
||||
|
||||
## Доступ к полному тексту
|
||||
|
||||
Требование авторизации, подписки или покупки онлайн-доступа означает недоступность книги. Аннотация, оглавление и ознакомительный фрагмент не подтверждают доступ к полному тексту. HTTP200 и Content-Type PDF сами по себе не дают зелёный статус; при недостатке доказательств возвращается `unknown`. Даже полученный PDF может быть фрагментом издания.
|
||||
|
||||
Правила учитывают сообщения читалок Лани, Юрайта и iBooks, изъятие публикации и предложение покупки онлайн-доступа Znanium, покупку именно eBook Springer и гостевое ограничение BOOK.ru. Цена печатной книги и общий запрет скачивания PDF не доказывают закрытое чтение. Результат содержит короткое доказательство, URL без токенов, время, версию правил и профиль `anonymous`.
|
||||
|
||||
Страницы известных ЭБС открываются в Chromium/Playwright для подтверждения видимых ограничений: скрытые CSS-шаблоны не считаются доказательством; проверяется видимый текст, при необходимости адрес читалки, а у BOOK.ru — подсказка к гостевому просмотру. Для остальных сайтов неопределённость сохраняется. Проверка не выполняет вход, покупку и обход CAPTCHA.
|
||||
|
||||
Docker включает Chromium с включённой песочницей, UID1000, read-only rootfs, no-new-privileges и cap-drop ALL. Профиль seccomp разрешает необходимые для песочницы user namespaces, остальные ограничения остаются; источник и лицензия указаны в deploy/README.md. Браузер получает минимальное окружение без DATABASE_URL. Все HTTP-запросы перехватываются, DNS проверяется и закрепляется в Node-сокете. Прямой выход блокируется нерабочим proxy, WebSocket/SW отключены, непроходящий через proxy WebRTC UDP запрещён. Методы записи, внешние домены, изображения/медиа/шрифты блокируются. Лимиты:100 запросов,32MB на задачу,8MB на ресурс,10 одновременных запросов; browser deadline30s и общий link deadline60s. Ошибка/лимит оставляет доступ неподтверждённым.
|
||||
|
||||
Локальная разработка браузерной части рекомендована в Docker с тем же seccomp-профилем. `pnpm test:browser` с локальной изолированной TEST_DATABASE_URL проверяет реальный Chromium и запрет обращений к локальному HTTP/UDP серверу через fetch, redirect, iframe, popup, worker, WebSocket и WebRTC. Обычные `pnpm test` также проверяют DNS/IP pinning и правила провайдеров.
|
||||
|
||||
## Частота запросов
|
||||
|
||||
Материалы проверяются последовательно. Между запросами к одному хосту выдерживается 2 секунды, включая каталоги, курсы и перенаправления. В браузерной проверке пауза применяется к навигациям; JS/CSS и anonymous API загружаются с ограничением параллелизма, без двухсекундной паузы на каждый ресурс. Ограничение общее для API и worker и сохраняется в PostgreSQL. После HTTP429/503 действует пауза из `Retry-After` (не менее минуты), после HTTP403 — пять минут. Повторных попыток в цикле нет. Ссылки, пропущенные во время паузы, отмечаются как ограниченные с датой окончания паузы; это не подтверждает, что материал удалён. Их можно обновить последующей полной проверкой. Статическая загрузка ограничена30 секундами, вся проверка ссылки с браузером —60 секундами. Proxy read timeout85s оставляет время на сохранение результата в пределах90s lease. Паузы снижают нагрузку, но не гарантируют отсутствие блокировки со стороны сайта.
|
||||
|
||||
## Расписание
|
||||
|
||||
В интерфейсе откройте «Проверка по расписанию»: включите автоматический запуск, выберите ежедневное или еженедельное повторение, время по Москве, при необходимости день недели и режим кэша. Сохранение и отключение доступны там же. По умолчанию расписание выключено; оно общее для всех посетителей и хранится в той же базе PostgreSQL.
|
||||
|
||||
`pnpm dev`, `pnpm start` и Docker запускают веб-сервер и отдельный фоновый процесс одним управляемым runtime. Проверка по расписанию выполняется без открытой вкладки, при перезапуске продолжается по сохранённой очереди. Расписание проверяется раз в пять секунд. Ручные проверки сохраняют прежний режим с открытой вкладкой.
|
||||
|
||||
Если любая предыдущая проверка ещё имеет статус running, совпавший слот пропускается с отметкой в интерфейсе: параллельных запусков нет. После простоя сервера выполняется один просроченный запуск; пропущенные дни не образуют очередь. Изменение или отключение расписания не прерывает уже начатую автоматическую проверку. Если один из двух процессов runtime завершится, второй останавливается; restart policy Docker поднимает их вместе.
|
||||
|
||||
## Production
|
||||
|
||||
```sh
|
||||
docker compose build
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
Эти команды изменяют production: выполнять только через разрешённый deployment-процесс после независимого review. Сервис слушает только `127.0.0.1:3011`, работает от UID1000 и использует существующий сервер PostgreSQL через внешнюю Docker-сеть `miem_workers_default`. Отдельный контейнер БД или SQLite volume не создаётся. Healthcheck: `GET /courses/api/health`. Порт3011 — предлагаемый; перед запуском проверить, что он свободен. Код не зависит от Cloudflare Workers, D1 или Sites hosting.
|
||||
|
||||
Добавить `deploy/nginx-courses.conf` в существующий TLS-server `miem.space`, проверить `nginx -t`, затем выполнить разрешённый reload. Конфиг передаёт исходный URI без удаления `/courses`; `/courses-other` не захватывается. Фактический путь Nginx-конфига должен быть подтверждён read-only preflight, а не угадан.
|
||||
|
||||
## Порядок первого deployment
|
||||
|
||||
1. Подтвердить актуальные branch/head, независимое review, тесты, сборку, Docker daemon/диск, свободный порт и отсутствие существующей локации `/courses`.
|
||||
2. Сохранить текущую конфигурацию Nginx. Использовать отдельную базу `miem_courses_links` и отдельную LOGIN-роль без SUPERUSER/CREATEDB/CREATEROLE. Не использовать учётную запись и таблицы существующих приложений. Перед изменением уже заполненной базы сделать pg_dump и проверить возможность восстановления; на первом размещении новая база пустая.
|
||||
3. Собрать exact commit, запустить только новый сервис. Проверить localhost healthcheck и `/courses`, статические файлы и шрифты.
|
||||
4. Включить Nginx location, выполнить syntax check/reload через разрешённый канал. Проверить HTTPS страницу, healthcheck и ручной шаг проверки, затем перезапуск сервиса с сохранением данных.
|
||||
5. При неудаче вернуть прежний Nginx-конфиг, проверить и reload; остановить только новый сервис. Базу и роль не удалять. При обновлении вернуть предыдущий совместимый image tag, сохранив PostgreSQL. Не применять `compose down -v` или общий prune.
|
||||
|
||||
Обновление Nginx обычно не требует остановки существующих сервисов; фактический риск и разрешённый способ reload определяются инфраструктурным runbook. DATABASE_URL передаётся Docker Compose из окружения либо через защищённый deployment env-file; его содержимое не публикуется и не коммитится. В этом checkout production deployment не выполнен.
|
||||
|
||||
## Проверки
|
||||
|
||||
`TEST_DATABASE_URL` должен вести только на локальный `sverka_test`; DB-тесты создают и удаляют собственные случайные схемы и отказываются работать с другой базой. `pnpm test` проверяет извлечение ссылок, классификацию, URL со скобками, запрет непубличных IP, атомарный rollback/replay очереди, единственный общий запуск, lease, создание/изменение/отключение расписания, московское время и переходы календаря, однократный claim, skip пересекающихся запусков и выполнение worker без браузера. Node HTTP-загрузчик закрепляет проверенный DNS-адрес в сокете и проверяет каждый redirect; TLS проверяет исходное имя хоста. Прокси из environment не используются загрузчиком.
|
||||
@@ -0,0 +1,15 @@
|
||||
import {dashboard,start,step} from '../../../lib/scanner.ts';
|
||||
import {saveSchedule} from '../../../lib/scheduler.ts';
|
||||
export const dynamic='force-dynamic';
|
||||
export const runtime='nodejs';
|
||||
export async function GET(){try{return Response.json(await dashboard(),{headers:{'Cache-Control':'no-store'}});}catch{return Response.json({error:'Не удалось прочитать результаты. Попробуйте обновить страницу.'},{status:503});}}
|
||||
export async function POST(request:Request){
|
||||
const origin=request.headers.get('origin');
|
||||
// Production uses a fixed public origin. Next's internal URL may use localhost
|
||||
// during development even when the browser connected to 127.0.0.1.
|
||||
const host=request.headers.get('host');
|
||||
const expectedOrigin=process.env.COURSES_ORIGIN??(process.env.NODE_ENV!=='production'&&host?new URL(request.url).protocol+'//'+host:null);
|
||||
if(!origin||!expectedOrigin||origin!==expectedOrigin)return Response.json({error:'Запрос с другого сайта запрещён'},{status:403});
|
||||
try{const body=await request.json() as {action?:string;force?:boolean;schedule?:unknown};if(body.action==='schedule'){
|
||||
try{return Response.json(await saveSchedule(body.schedule));}catch(error){if(error instanceof RangeError)return Response.json({error:error.message},{status:400});throw error;}
|
||||
}if(body.action==='start')return Response.json(await start(body.force===true));if(body.action==='step')return Response.json(await step());return Response.json({error:'Неизвестное действие'},{status:400});}catch{return Response.json({error:'Не удалось выполнить шаг. Сохранённый прогресс можно продолжить.'},{status:503});}}
|
||||
@@ -0,0 +1,5 @@
|
||||
import {database} from '../../../lib/store';
|
||||
import {VERSION} from '../../../lib/model';
|
||||
export const dynamic='force-dynamic';
|
||||
export const runtime='nodejs';
|
||||
export async function GET(){try{await (await database()).query('SELECT 1');return Response.json({status:'ok',version:VERSION},{headers:{'Cache-Control':'no-store'}});}catch{return Response.json({status:'error'},{status:503});}}
|
||||
@@ -0,0 +1,35 @@
|
||||
'use client';
|
||||
import {useEffect,useRef,useState} from 'react';
|
||||
import {ArrowUpRight,Download,BookOpen,Check,ChevronDown,ExternalLink,FileSearch,Link2,LoaderCircle,Pause,Play,Search,ShieldCheck,SlidersHorizontal,X,RefreshCw} from 'lucide-react';
|
||||
import SchedulePanel from './schedule-panel';
|
||||
import {PROGRAMS,LABELS,VERSION,DEFAULT_SCHEDULE,type Dashboard,type Run} from '../lib/model';
|
||||
const date=(v?:string)=>v?new Date(v).toLocaleString('ru-RU',{timeZone:'Europe/Moscow',day:'2-digit',month:'short',hour:'2-digit',minute:'2-digit'}):'Ещё не проверено';
|
||||
export default function Audit(){
|
||||
const [data,setData]=useState<Dashboard>({run:null,courses:[],checks:{},programs:PROGRAMS,schedule:{...DEFAULT_SCHEDULE}}),[loaded,setLoaded]=useState(false);
|
||||
const [loading,setLoading]=useState(true),[active,setActive]=useState(false),[error,setError]=useState(''),[program,setProgram]=useState('all'),[query,setQuery]=useState(''),[status,setStatus]=useState('issues'),[view,setView]=useState('links'),[force,setForce]=useState(false),[expanded,setExpanded]=useState(''),[limit,setLimit]=useState(40);
|
||||
const running=useRef(false),mounted=useRef(true);
|
||||
async function refresh(){const r=await fetch('/courses/api/audit',{cache:'no-store'}),d=await r.json() as Dashboard & {error?:string};if(!r.ok)throw new Error(d.error);if(mounted.current){setData(d);setLoaded(true);}}
|
||||
useEffect(()=>{mounted.current=true;refresh().catch(e=>setError(e.message)).finally(()=>setLoading(false));return()=>{mounted.current=false;running.current=false;};},[]);
|
||||
async function post(action:string){const r=await fetch('/courses/api/audit',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({action,force})}),d=await r.json() as Run & {error?:string};if(!r.ok)throw new Error(d.error);return d as Run;}
|
||||
async function launch(){if(running.current)return;running.current=true;setActive(true);setError('');try{let run=data.run?.status==='running'?data.run:await post('start');setData(d=>({...d,run}));while(running.current&&run?.status==='running'){run=await post('step');if(!mounted.current)break;setData(d=>({...d,run}));await refresh();await new Promise(r=>setTimeout(r,350));}}catch(e){setError((e as Error).message);}finally{running.current=false;if(mounted.current){setActive(false);refresh().catch(()=>{});}}}
|
||||
function pause(){running.current=false;}
|
||||
const serverActive=data.run?.source==='schedule'&&data.run.status==='running';
|
||||
useEffect(()=>{if(active||(!data.schedule.enabled&&!serverActive))return;const timer=setInterval(()=>{refresh().catch(e=>setError(e.message));},3000);return()=>clearInterval(timer);},[active,data.schedule.enabled,serverActive]);
|
||||
const run=data.run,courses=data.courses.filter(c=>program==='all'||c.program===program);
|
||||
const rows=courses.flatMap(course=>course.refs.map(ref=>({course,ref,result:data.checks[ref.url]}))),problems=rows.filter(r=>r.result&&r.result.status!=='ok');
|
||||
const visible=rows.filter(r=>(status==='all'||status==='issues'&&r.result&&r.result.status!=='ok'||status==='pending'&&!r.result||r.result?.status===status)&&(!query||(r.ref.title+' '+r.ref.url+' '+r.course.name).toLowerCase().includes(query.toLowerCase())));
|
||||
const shownCourses=courses.filter(c=>!query||(c.name+' '+c.url).toLowerCase().includes(query.toLowerCase()));
|
||||
useEffect(()=>setLimit(40),[query,status,program,view]);
|
||||
function exportCsv(){const rows=[['Программа','Курс','Ссылка на курс','Литература','URL','Статус','Причина','Проверено'],...visible.map(r=>[PROGRAMS.find(p=>p.id===r.course.program)?.name??'',r.course.name,r.course.url,r.ref.title,r.ref.url,r.result?LABELS[r.result.status]:'Не проверено',r.result?.reason??'',r.result?.checkedAt??''])];const csv='\uFEFF'+rows.map(row=>row.map(v=>'"'+String(v).replace(/^[=+@-]/,"'$&").replaceAll('"','""')+'"').join(';')).join('\r\n');const a=document.createElement('a');a.href=URL.createObjectURL(new Blob([csv],{type:'text/csv;charset=utf-8'}));a.download='sverka-links.csv';a.click();setTimeout(()=>URL.revokeObjectURL(a.href),1000);}
|
||||
const selected=PROGRAMS.find(p=>p.id===program),percent=run?.phase==='links'?Math.round(run.checked/Math.max(1,run.totalLinks)*100):run?.phase==='courses'?Math.round(run.processed/Math.max(1,run.totalCourses)*100):0;
|
||||
return <div className="shell"><a className="skip" href="#main">К результатам</a><aside className="sidebar"><a href="/courses" className="brand"><span className="brand__mark"><Link2 size={25}/></span>сверка<span className="brand__dot">.</span></a><div className="sidebar__description">Доступность знаний<br/>начинается со ссылки.</div><nav aria-label="Разделы" className="primary-nav"><button className={view==='links'?'selected':''} onClick={()=>setView('links')}><FileSearch size={18}/>Результаты проверки<span>{problems.length||'—'}</span></button><button className={view==='courses'?'selected':''} onClick={()=>setView('courses')}><BookOpen size={18}/>Учебные курсы</button></nav><div className="sidebar__label">Образовательные программы</div><nav className="program-nav" aria-label="Программы"><button className={program==='all'?'selected':''} onClick={()=>setProgram('all')}><span>Все программы</span><b>5</b></button>{PROGRAMS.map(p=><button key={p.id} className={program===p.id?'selected':''} onClick={()=>{setProgram(p.id);if(view==='method')setView('links');}}><span>{p.name}<small>{p.level}</small></span><ArrowUpRight size={15}/></button>)}</nav><div className="sidebar__bottom"><ShieldCheck size={20}/><p>Без регистрации.<br/><span>Общий реестр результатов.</span></p></div></aside><div className="workspace"><header className="topbar"><span>НИУ ВШЭ <span className="topbar__divider">/</span> Учебная литература</span><button onClick={()=>setView(view==='method'?'links':'method')}>Как это работает <ArrowUpRight size={14}/></button></header><main id="main"><div className="page-title"><div><h1>{view==='method'?'Как устроена сверка':view==='courses'?'Учебные курсы':'Ссылки под наблюдением'}</h1><p>{view==='method'?'Что проверяется и как читать результаты.':'Проверяйте литературу. Находите недоступные материалы.'}</p></div><span className="edition">5 программ<span>единый реестр</span></span></div>
|
||||
{error&&<div role="alert" className="error-banner"><span>{error}</span><button onClick={()=>{setError('');refresh().catch(e=>setError(e.message));}}>Повторить</button></div>}
|
||||
{view==='method'?<section className="method"><h2>От программы — к материалу</h2><p>Сверка читает текущий учебный год на пяти страницах ВШЭ, обходит пагинацию и открывает каждый курс. Из разделов основной и дополнительной литературы извлекаются кликабельные ссылки и адреса, записанные обычным текстом.</p><h2>Вы управляете проверкой</h2><p>Нажмите «Запустить проверку» и оставьте вкладку открытой. Можно приостановить проверку, закрыть вкладку и позже продолжить сохранённый запуск. Одновременно выполняется один общий запуск. Приостановка в этой вкладке не останавливает проверку у другого посетителя.</p><h2>Проверка по расписанию</h2><p>Настройте ежедневный или еженедельный запуск и время по Москве. Расписание общее для всех посетителей и сохраняется на сервере. Автоматическая проверка работает без браузера. Изменение или отключение расписания влияет на будущие запуски; уже начатая проверка продолжается. Если другая проверка ещё идёт, совпавший запуск пропускается. После простоя сервера выполняется один просроченный запуск, без очереди пропущенных повторений.</p><h2>Изменения вместо повторений</h2><p>Страницы курсов сравниваются по контрольной сумме. Неизменившийся список литературы используется повторно. Доступность ссылок имеет отдельный срок кэша: сутки для подтверждённых ограничений и десять минут для неопределённых результатов или временных ошибок. После изменения правил ссылки проверяются заново. Полная проверка игнорирует кэш.</p><h2>Проверяется доступ без авторизации</h2><p>Различаются 404/410, вход, ограничения доступа, CAPTCHA, ошибки сервера и тайм-ауты. Если для чтения требуется авторизация, подписка или покупка, книга считается недоступной. Причина подтверждается сообщением сайта или предложением покупки онлайн-доступа. Карточка, аннотация и фрагмент не подтверждают доступ ко всей книге. Для поддерживаемых ЭБС проверяется содержимое после загрузки JavaScript и, при необходимости, читалка. При недостатке доказательств показывается «Доступ не подтверждён». Сверка не обходит вход и защиту сайта.</p><h2>Границы проверки</h2><p>Проверяется литература в HTML-программах. Если раздел отсутствует, доступен только во вложении или не читается, курс получает отдельную отметку. Архивные годы и навигационные ссылки не проверяются. Старые результаты сохраняются с датами; непрочитанные и исчезнувшие курсы отмечаются как не обновлённые.</p></section>:<>
|
||||
<section className="control" aria-label="Управление проверкой"><div className="control__icon"><Link2 size={25}/></div><div className="control__copy"><h2>{serverActive?'Выполняется проверка по расписанию':active?'Сверяем источники':run?.status==='running'?'Проверка ожидает продолжения':run?'Результаты сохранены':'Начните с первой проверки'}</h2><p aria-live="polite">{active||serverActive?run?.message:run?`Последний запуск: ${date(run.startedAt)} · МСК`:'Все курсы пяти программ. Только реальные результаты.'}</p></div><div className="control__actions">{serverActive?<button disabled className="button button--primary"><LoaderCircle size={16} className="spin"/>Выполняется на сервере</button>:active?<button className="button button--primary" onClick={pause}><Pause size={16}/>Приостановить</button>:<button disabled={loading} className="button button--primary" onClick={launch}>{loading?<LoaderCircle size={16} className="spin"/>:<Play size={16}/>} {run?.status==='running'?'Продолжить проверку':'Запустить проверку'}</button>}<label className="force"><input type="checkbox" checked={force} disabled={active||run?.status==='running'} onChange={e=>setForce(e.target.checked)}/>Полная проверка без кэша</label></div>{run?.status==='running'&&<div className="scan-progress"><div><span>{run.phase==='catalogs'?`Каталоги: ${run.visited.length}`:run.phase==='courses'?`Курсы: ${run.processed} / ${run.totalCourses}`:`Адреса: ${run.checked} / ${run.totalLinks}`}</span><span>{serverActive?'Выполняется без открытой вкладки':'Оставьте вкладку открытой'}</span></div><progress aria-label="Ход проверки" max={100} value={run.phase==='catalogs'?undefined:percent}/></div>}</section>
|
||||
{loaded?<SchedulePanel schedule={data.schedule} onSaved={schedule=>setData(d=>({...d,schedule}))}/>:<p className="schedule__hint" role="status">{loading?'Загружаем настройки расписания…':'Расписание недоступно. Повторите загрузку через сообщение об ошибке выше.'}</p>}<div className="summary" aria-label="Сводка"><div><span className="summary__number summary__number--problem">{problems.length||'—'}</span><span>ссылок требуют внимания</span></div><div><span className="summary__number">{courses.length||'—'}</span><span>курсов в реестре</span></div><div><span className="summary__number">{rows.filter(r=>r.result?.status==='ok').length||'—'}</span><span>полных текстов доступны</span></div><div><span className="summary__number">{run?.reused??'—'}</span><span>курсов взято из кэша</span></div></div>
|
||||
{run&&run.errors.length>0&&<details className="source-errors"><summary>Не удалось прочитать {run.errors.length} источников — охват неполный</summary>{run.errors.map((e,i)=><p key={i}><a href={e.url.startsWith('https://')?e.url:undefined} target="_blank" rel="noreferrer">{e.url}</a><br/>{e.reason}</p>)}</details>}
|
||||
<section className="results"><div className="results__heading"><h2>{view==='courses'?'Каталог курсов':'Результаты проверки'}<span>{view==='courses'?shownCourses.length:visible.length}</span></h2><button className="text-button" onClick={exportCsv} disabled={!visible.length}><Download size={16}/>Экспорт CSV</button></div><div className="filters"><label className="search"><Search size={18}/><input aria-label="Поиск по курсу, материалу или адресу" placeholder="Найти курс, материал или ссылку" value={query} onChange={e=>setQuery(e.target.value)}/>{query&&<button aria-label="Очистить поиск" onClick={()=>setQuery('')}><X size={16}/></button>}</label>{view==='links'&&<label className="status-select"><SlidersHorizontal size={16}/><select aria-label="Причина недоступности" value={status} onChange={e=>setStatus(e.target.value)}><option value="issues">Требуют внимания</option><option value="all">Все ссылки</option>{Object.entries(LABELS).map(([k,v])=><option value={k} key={k}>{v}</option>)}<option value="pending">Не проверены</option></select><ChevronDown size={14}/></label>}</div>{selected&&<div className="scope">{selected.name}<button aria-label="Сбросить программу" onClick={()=>setProgram('all')}><X size={14}/></button><a href={selected.url} target="_blank" rel="noreferrer">Каталог ВШЭ <ExternalLink size={13}/></a></div>}
|
||||
{loading?<div className="empty" role="status"><LoaderCircle className="spin" size={30}/><h3>Загружаем реестр</h3></div>:view==='courses'?shownCourses.length?<div className="course-list">{shownCourses.slice(0,limit).map(c=><article key={c.url}><div><a className="course-name" href={c.url+'#library'} target="_blank" rel="noreferrer">{c.name}<ArrowUpRight size={16}/></a><p>{PROGRAMS.find(p=>p.id===c.program)?.name}</p>{c.issue&&<p className="course-issue">{c.issue}</p>}<small>{date(c.checkedAt)} · {c.refs.length} адресов {c.reused?'· из кэша':''}{run&&c.seen!==run.id?' · не обновлён в текущем запуске':''}</small></div><span className={'badge '+(c.issue?'badge--blocked':'badge--ok')}>{c.issue?'Нужна проверка':c.refs.length?'Литература найдена':'Нет ссылок'}</span></article>)}</div>:<Empty hasRun={!!run} filtered={!!query||program!=='all'} launch={launch}/>:visible.length?<><div className="table-head"><span>Материал и источник</span><span>Результат</span><span>Проверено, МСК</span></div><div className="link-list">{visible.slice(0,limit).map(r=>{const key=r.course.url+r.ref.url;return <article key={key} className="link-row"><div className="link-row__main"><span className={'status-dot status-dot--'+(r.result?.status??'pending')}/><div><a className="material-name" href={r.ref.url} target="_blank" rel="noreferrer">{r.ref.title||r.ref.url}<ArrowUpRight size={15}/></a><div className="material-url">{r.ref.url}</div><a className="source-course" href={r.course.url+'#library'} target="_blank" rel="noreferrer"><BookOpen size={13}/>{r.course.name}<ArrowUpRight size={12}/></a><small className="program-name">{PROGRAMS.find(p=>p.id===r.course.program)?.name}{run&&r.course.seen!==run.id?' · прежний результат':''}</small></div></div><div className="link-row__result"><span className={'badge badge--'+(r.result?.status??'pending')}>{r.result?LABELS[r.result.status]:'Не проверено'}</span>{r.result&&['auth','restricted'].includes(r.result.status)&&<p className="link-row__reason">{r.result.reason}</p>}<button className="reason-toggle" aria-expanded={expanded===key} onClick={()=>setExpanded(expanded===key?'':key)}>Причина <ChevronDown size={12}/></button></div><div className="link-row__date">{date(r.result?.checkedAt)}</div>{expanded===key&&<div className="reason"><p>{r.result?.reason??'Адрес ещё не проверен. Запустите или продолжите сверку.'}</p>{r.result?.evidence&&<div className="reason__evidence"><blockquote>{r.result.evidence.text}</blockquote><a href={r.result.evidence.url} target="_blank" rel="noreferrer">Сообщение на сайте <ExternalLink size={12}/></a><small>Проверено без авторизации</small></div>}{r.result?.code&&<small>Технические сведения: HTTP {r.result.code}</small>}{r.result?.finalUrl&&r.result.finalUrl!==r.ref.url&&<a href={r.result.finalUrl} target="_blank" rel="noreferrer">После перенаправления: {r.result.finalUrl}</a>}{r.course.reused&&<small>Список литературы курса не изменился. Доступность ссылок проверяется отдельно.</small>}</div>}</article>;})}</div></>:<Empty hasRun={!!run} filtered={!!query||program!=='all'||status!=='issues'} launch={launch}/>}
|
||||
{(view==='courses'?shownCourses.length:visible.length)>limit&&<button className="load-more" onClick={()=>setLimit(n=>n+40)}>Показать ещё 40</button>}<div className="results__note"><ShieldCheck size={15}/><span>Блокировка проверки не означает, что материал удалён. Причину можно уточнить по исходной ссылке.</span></div></section><div className="bottom-note"><span>Проверяем изменения, сохраняем результаты.</span><span><RefreshCw size={13}/>Неизменившиеся курсы — из кэша</span></div></>}
|
||||
<footer><span>сверка. <span>Независимый инструмент · источники: hse.ru</span></span><span>Интерфейс {VERSION} · Сервер {VERSION}</span></footer></main></div></div>;}
|
||||
function Empty({hasRun,filtered,launch}:{hasRun:boolean;filtered:boolean;launch:()=>void}){return <div className="empty"><div className="empty__drawing"><FileSearch size={43} strokeWidth={1.15}/><span><Check size={15}/></span></div><h3>{filtered?'По этим условиям ничего нет':hasRun?'Проблемных ссылок пока нет':'Пока ни одна ссылка не проверена'}</h3><p>{filtered?'Измените запрос, причину или выбранную программу.':hasRun?'Смотрите охват и список курсов. Непрочитанные источники отмечаются отдельно.':'Запустите сверку, чтобы найти недоступную литературу и увидеть, в каких курсах она используется.'}</p>{!hasRun&&!filtered&&<button className="text-button" onClick={launch}>Проверить пять программ<ArrowUpRight size={16}/></button>}<div className="empty__legend"><span><i className="status-dot status-dot--broken"/>404 / удалена</span><span><i className="status-dot status-dot--auth"/>Требуется вход</span><span><i className="status-dot status-dot--blocked"/>Ограничен доступ</span></div></div>;}
|
||||
@@ -0,0 +1,12 @@
|
||||
@import url('/courses/fonts/fonts.css');
|
||||
:root{--paper:#f7f7f2;--white:#fff;--ink:#243a32;--muted:#69756d;--forest:#203e32;--line:#dfe3da;--red:#a54735;--accent:#d9ebbb;--sans:'Golos Text',sans-serif;--serif:'Lora',Georgia,serif}
|
||||
*{box-sizing:border-box}body{margin:0;color:var(--ink);background:var(--paper);font:14px/1.55 var(--sans);-webkit-font-smoothing:antialiased}button,input,select{font:inherit}button,a,input,select{touch-action:manipulation}button{cursor:pointer;color:inherit}button:disabled{opacity:.45;cursor:default}a{color:inherit;text-decoration:none;text-underline-offset:4px}a:hover{text-decoration:underline}button{border:0;background:none}button:focus-visible,a:focus-visible,input:focus-visible,select:focus-visible,summary:focus-visible{outline:2px solid #85a457;outline-offset:4px}::selection{background:#d9ebbb;color:#203e32}input{caret-color:var(--forest)}::-webkit-scrollbar{width:8px;height:8px}::-webkit-scrollbar-thumb{background:#b7c2b5;border-radius:8px}svg{flex-shrink:0;vertical-align:middle}.shell{min-height:100vh;display:grid;grid-template-columns:275px minmax(0,1fr)}.sidebar{background:var(--forest);color:#f2f5ed;padding:38px 21px 24px;display:flex;flex-direction:column;position:sticky;top:0;height:100vh}.brand{display:flex;align-items:center;font-size:32px;font-weight:600;letter-spacing:-1.3px;padding:0 10px;line-height:1}.brand:hover{text-decoration:none}.brand__mark{border:1px solid #b9ccad;width:38px;height:38px;display:grid;place-items:center;border-radius:50%;margin-right:11px;color:var(--accent)}.brand__dot{color:var(--accent)}.sidebar__description{color:#c1cfbf;font-size:12px;line-height:1.7;margin:24px 10px 39px}.primary-nav{display:grid;gap:6px}.primary-nav button{display:flex;align-items:center;text-align:left;gap:11px;padding:13px 11px;font-size:12px;border-radius:5px;color:#d6dfd1}.primary-nav button.selected{background:#3d5743;color:#f6f9f0}.primary-nav button:hover,.program-nav button:hover{background:#35513e}.primary-nav button span{margin-left:auto;background:#50664c;color:#e0edce;border-radius:3px;padding:0 6px;font-size:11px}.sidebar__label{font-size:10px;text-transform:uppercase;letter-spacing:1px;color:#b6c6af;margin:37px 10px 14px}.program-nav{display:grid;gap:3px}.program-nav button{text-align:left;color:#d9e3d3;display:flex;align-items:flex-start;justify-content:space-between;gap:9px;font-size:12px;line-height:1.55;padding:12px 11px;border-radius:5px}.program-nav button.selected{background:#d9ebbb;color:#223c2e}.program-nav small{display:block;font-size:10px;opacity:.75;margin-top:4px}.program-nav svg{margin-top:2px}.program-nav b{font-weight:500}.sidebar__bottom{border-top:1px solid #4b604a;display:flex;gap:12px;align-items:center;margin: auto 10px 0;padding-top:20px;color:#c5d5b9}.sidebar__bottom p{font-size:11px;line-height:1.7;margin:0}.sidebar__bottom span{color:#b2c1a9}.workspace{min-width:0}.topbar{height:76px;border-bottom:1px solid var(--line);display:flex;align-items:center;justify-content:space-between;padding:0 44px;font-size:11px}.topbar>span{font-weight:500}.topbar__divider{color:#a7b3a2;margin:0 14px}.topbar button{display:flex;gap:10px;align-items:center;color:var(--muted);font-size:11px}main{max-width:1530px;margin:0 auto;padding:44px 44px 0}.page-title{display:flex;justify-content:space-between;gap:20px;align-items:center;margin:0 0 32px}h1{font-family:var(--serif);font-weight:400;font-size:clamp(28px,3vw,43px);line-height:1.2;letter-spacing:-1.3px;margin:0 0 12px}.page-title p{margin:0;color:var(--muted);font-size:12px}.edition{font-size:12px;white-space:nowrap;text-align:right;padding-left:22px;border-left:1px solid var(--line)}.edition span{display:block;color:var(--muted);font-size:10px;margin-top:4px}.control{border:1px solid #ced8c2;display:grid;grid-template-columns:auto minmax(0,1fr) auto;align-items:center;gap:17px;background:#eef2e7;padding:26px 25px;border-radius:7px}.control__icon{width:47px;height:47px;display:grid;place-items:center;background:#e0e8d5;border-radius:50%;color:#4b6340}.control h2{font-size:15px;margin:0 0 6px;font-weight:550}.control p{font-size:11px;color:#5f7058;margin:0;overflow-wrap:anywhere}.control__actions{display:grid;gap:10px;justify-items:center}.button{display:inline-flex;align-items:center;justify-content:center;gap:9px;border-radius:5px;padding:12px 16px;font-size:12px;font-weight:500;white-space:nowrap;transition:background .15s}.button--primary{background:var(--forest);color:#f4f7ef}.button--primary:hover{background:#345b42}.force{display:flex;align-items:center;font-size:10px;color:#596950;gap:6px;cursor:pointer}.force input{accent-color:#34553a;width:12px;height:12px;margin:0}.scan-progress{grid-column:1/-1;border-top:1px solid #d4dfcb;padding-top:14px;font-size:11px;color:#536a48}.scan-progress>div{display:flex;justify-content:space-between;gap:10px}progress{width:100%;height:5px;accent-color:var(--forest);margin-top:10px;display:block}.summary{display:grid;grid-template-columns:1.2fr 1fr 1fr 1fr;padding:30px 0 28px;border-bottom:1px solid var(--line);gap:18px;margin-bottom:34px}.summary>div{display:flex;flex-direction:column;padding-left:24px;border-left:1px solid var(--line)}.summary>div:first-child{padding-left:0;borderLine truncated
|
||||
@media(min-width:1500px){.shell{grid-template-columns:290px minmax(0,1fr)}main{padding:55px 64px 0}.topbar{padding:0 64px}.sidebar{padding-top:42px}.control{padding:30px}.summary{padding:35px 0}.empty{min-height:340px}}
|
||||
@media(max-width:1150px){.shell{grid-template-columns:235px minmax(0,1fr)}.sidebar{padding:28px 13px 20px}.sidebar__description{margin-bottom:28px}.sidebar__label{font-size:9px}.program-nav button{font-size:11px;padding:10px}.primary-nav button{font-size:11px}.topbar{padding:0 27px}main{padding:35px 27px 0}.control{grid-template-columns:auto minmax(0,1fr);padding:20px}.control__actions{grid-column:2;justify-items:start}.edition{display:none}.summary{gap:10px}.summary>div{padding-left:14px}.table-head,.link-row{grid-template-columns:minmax(0,1fr) 130px}.link-row__date{grid-column:2}.table-head>span:last-child{display:none}.filters{flex-wrap:wrap}.search{flex-basis:260px}footer{flex-wrap:wrap;gap:8px}.sidebar__bottom{margin-top:25px}.sidebar{overflow-y:auto}.bottom-note{flex-wrap:wrap}}
|
||||
@media(max-width:760px){.shell{display:block}.sidebar{height:auto;position:relative;padding:22px 20px 14px;overflow:visible}.brand{font-size:28px;padding:0}.brand__mark{width:32px;height:32px}.sidebar__description,.sidebar__label,.sidebar__bottom{display:none}.primary-nav{display:flex;margin-top:23px;gap:10px}.primary-nav button{font-size:11px;padding:10px;flex:1;justify-content:center}.program-nav{display:flex;overflow-x:auto;gap:7px;margin-top:15px;padding-bottom:5px}.program-nav button{white-space:nowrap;flex-shrink:0;padding:7px 11px;max-width:235px;overflow:hidden;align-items:center;background:#2d4c38;font-size:10px}.program-nav button>span{overflow:hidden;text-overflow:ellipsis}.program-nav small,.program-nav svg{display:none}.program-nav b{margin-left:4px}.topbar{height:54px;padding:0 20px;font-size:9px}.topbar__divider{margin:0 8px}.topbar button{font-size:9px;gap:5px}main{padding:28px 20px 0}.page-title{margin-bottom:23px}h1{font-size:31px;letter-spacing:-.8px}.page-title p{font-size:11px;max-width:290px;line-height:1.8}.control{gap:12px;padding:19px 16px;grid-template-columns:auto minmax(0,1fr)}.control__icon{width:38px;height:38px}.control h2{font-size:13px}.control p{font-size:10px}.control__actions{grid-column:1/-1;justify-items:stretch;margin-top:6px}.force{justify-content:center}.summary{grid-template-columns:1fr 1fr;gap:23px 10px;margin-bottom:25px;padding:25px 0}.summary>div{padding-left:17px}.summary>div:nth-child(3){padding-left:0;border:0}.summary__number{font-size:27px}.summary>div>span:last-child{font-size:10px}.results__heading{padding:18px 14px 15px;gap:6px}.results__heading h2{font-size:14px;gap:7px}.text-button{font-size:10px;gap:5px}.filters{padding:0 14px 16px;gap:8px}.search{padding:10px;font-size:10px}.status-select{width:100%}.status-select select{flex:1;max-width:none}.table-head{display:none}.link-row{padding:18px 14px;grid-template-columns:minmax(0,1fr) auto;gap:8px 12px}.link-row__main{grid-column:1/-1}.link-row__result{grid-column:1;padding-left:19px;display:flex;gap:10px;align-items:center}.link-row__date{grid-column:2;font-size:9px}.link-row__date small{display:none}.material-name{font-size:12px}.program-name{font-size:9px}.empty{padding:29px 16px 24px;min-height:320px}.empty h3{font-size:15px}.empty p{font-size:11px}.empty__legend{gap:13px;font-size:9px}.results__note{font-size:9px;padding:12px 14px;align-items:start}.results__note svg{margin-top:2px}.bottom-note{font-size:9px}.course-list article{padding:17px 14px;flex-direction:column;gap:9px}.course-list .badge{align-self:flex-start}.scope{padding:0 14px 15px;flex-wrap:wrap;font-size:10px}.scope a{margin-left:0}footer{margin-top:35px;padding:18px 0;font-size:9px;display:grid}footer>span>span{margin-left:6px;font-size:8px}.scan-progress>div{flex-direction:column;gap:4px}.error-banner{font-size:11px}.method h2{font-size:23px}.method p{font-size:13px}}
|
||||
@media(prefers-reduced-motion:reduce){*,*::before,*::after{animation:none!important;transition:none!important}}
|
||||
.schedule{margin-top:16px;border-bottom:1px solid var(--line)}.schedule summary{display:flex;align-items:center;gap:12px;cursor:pointer;list-style:none;padding:17px 2px;color:var(--forest)}.schedule summary::-webkit-details-marker{display:none}.schedule__title{font-size:13px;font-weight:500}.schedule__title small{display:block;font-size:10px;font-weight:400;color:var(--muted);margin-top:3px}.schedule__state{margin-left:auto;font-size:10px;color:var(--muted);white-space:nowrap}.schedule__state--on{color:#42602f}.schedule__chevron{transition:transform .15s}.schedule details[open] .schedule__chevron{transform:rotate(180deg)}.schedule__form{padding:8px 0 24px}.schedule__toggle{display:flex;align-items:center;gap:9px;font-size:12px;font-weight:500;cursor:pointer}.schedule input[type=checkbox]{accent-color:var(--forest);width:15px;height:15px;margin:0}.schedule__fields{display:flex;flex-wrap:wrap;gap:16px;margin:20px 0 14px}.schedule__fields label{display:flex;flex-direction:column;gap:7px;font-size:10px;color:var(--muted);min-width:140px;flex:1}.schedule__fields select,.schedule__fields input{border:1px solid #c9d2c3;border-radius:4px;padding:11px 10px;background:var(--white);color:var(--ink);font-size:12px;min-width:0;width:100%;height:42px}.schedule__hint{font-size:11px;line-height:1.8;color:var(--muted);max-width:80ch;margin:10px 0}.schedule__history{font-size:11px;margin:12px 0;color:#52654b}.schedule__error{font-size:11px;line-height:1.7;color:var(--red);margin:12px 0;overflow-wrap:anywhere}.schedule__actions{display:flex;flex-wrap:wrap;gap:16px;align-items:center;margin-top:19px}.schedule__actions>span{display:flex;align-items:center;gap:7px;color:#52654b;font-size:11px}@media(max-width:760px){.schedule summary{gap:9px}.schedule__title{font-size:12px}.schedule__title small{font-size:10px;max-width:230px}.schedule__state{font-size:9px}.schedule__fields{gap:12px}.schedule__fields label{flex-basis:calc(50% - 12px);min-width:120px}.schedule__actions{gap:12px}.schedule__actions .button{width:100%}}
|
||||
|
||||
.link-row__reason{font-size:10px;line-height:1.6;margin:6px 0 0;color:var(--ink);overflow-wrap:anywhere}.reason__evidence{margin-top:12px}.reason__evidence blockquote{margin:0;padding:10px 12px;border:1px solid var(--line);border-radius:4px;background:var(--paper);font-size:11px;line-height:1.7}
|
||||
|
||||
@media(max-width:760px){.link-row__result{grid-column:1/-1;flex-wrap:wrap}.link-row__reason{flex-basis:100%;order:3;margin-top:0}.link-row__date{grid-column:1/-1;padding-left:19px}}
|
||||
@@ -0,0 +1,23 @@
|
||||
import type { Metadata } from "next";
|
||||
import "./globals.css";
|
||||
|
||||
export const metadata: Metadata = {
|
||||
title: "Сверка — доступность учебной литературы",
|
||||
description: "Проверка ссылок на литературу в учебных курсах пяти программ НИУ ВШЭ.",
|
||||
icons: {
|
||||
icon: "/courses/favicon.svg",
|
||||
shortcut: "/courses/favicon.svg",
|
||||
},
|
||||
};
|
||||
|
||||
export default function RootLayout({
|
||||
children,
|
||||
}: Readonly<{
|
||||
children: React.ReactNode;
|
||||
}>) {
|
||||
return (
|
||||
<html lang="ru">
|
||||
<body className="antialiased">{children}</body>
|
||||
</html>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
import Audit from './dashboard';
|
||||
export default function Home(){return <Audit/>;}
|
||||
@@ -0,0 +1,30 @@
|
||||
'use client';
|
||||
import {useEffect,useState} from 'react';
|
||||
import {CalendarClock,ChevronDown,LoaderCircle,Check} from 'lucide-react';
|
||||
import {type Schedule,type ScheduleSettings} from '../lib/model';
|
||||
|
||||
const DAYS=['Воскресенье','Понедельник','Вторник','Среда','Четверг','Пятница','Суббота'];
|
||||
const date=(value:string|null)=>value?new Date(value).toLocaleString('ru-RU',{timeZone:'Europe/Moscow',day:'numeric',month:'long',hour:'2-digit',minute:'2-digit'}):'—';
|
||||
export default function SchedulePanel({schedule,onSaved}:{schedule:Schedule;onSaved:(schedule:Schedule)=>void}){
|
||||
const [form,setForm]=useState<ScheduleSettings>(schedule),[dirty,setDirty]=useState(false),[busy,setBusy]=useState(false),[error,setError]=useState(''),[saved,setSaved]=useState(false);
|
||||
useEffect(()=>{if(!dirty&&!busy)setForm(schedule);},[schedule,dirty,busy]);
|
||||
function update(value:Partial<ScheduleSettings>){setForm(f=>({...f,...value}));setDirty(true);setSaved(false);setError('');}
|
||||
async function save(event:React.FormEvent){event.preventDefault();setBusy(true);setError('');setSaved(false);
|
||||
try{const response=await fetch('/courses/api/audit',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({action:'schedule',schedule:{enabled:form.enabled,frequency:form.frequency,time:form.time,weekday:form.weekday,force:form.force}})});
|
||||
const result=await response.json() as Schedule & {error?:string};if(!response.ok)throw new Error(result.error??'Не удалось сохранить расписание.');setForm(result);setDirty(false);setSaved(true);onSaved(result);
|
||||
}catch(e){setError(e instanceof Error?e.message:'Не удалось сохранить расписание.');}finally{setBusy(false);}
|
||||
}
|
||||
return <section className="schedule" aria-label="Расписание проверки"><details><summary><CalendarClock size={19}/><span className="schedule__title">Проверка по расписанию<small>{schedule.enabled?`Следующий запуск: ${date(schedule.nextRunAt)} · МСК`:'Автоматическая проверка выключена'}</small></span><span className={'schedule__state '+(schedule.enabled?'schedule__state--on':'')}>{schedule.enabled?'Включено':'Настроить'}</span><ChevronDown className="schedule__chevron" size={15}/></summary>
|
||||
<form className="schedule__form" onSubmit={save}>
|
||||
<label className="schedule__toggle"><input type="checkbox" checked={form.enabled} onChange={e=>update({enabled:e.target.checked})} disabled={busy}/>Запускать проверку автоматически</label>
|
||||
<div className="schedule__fields"><label>Повторять<select value={form.frequency} onChange={e=>update({frequency:e.target.value as ScheduleSettings['frequency']})} disabled={busy}><option value="daily">Каждый день</option><option value="weekly">Каждую неделю</option></select></label>
|
||||
{form.frequency==='weekly'&&<label>День недели<select value={form.weekday} onChange={e=>update({weekday:Number(e.target.value)})} disabled={busy}>{[1,2,3,4,5,6,0].map(day=><option value={day} key={day}>{DAYS[day]}</option>)}</select></label>}
|
||||
<label>Время по Москве<input type="time" required value={form.time} onChange={e=>update({time:e.target.value})} disabled={busy}/></label>
|
||||
<label>Режим проверки<select value={String(form.force)} onChange={e=>update({force:e.target.value==='true'})} disabled={busy}><option value="false">Только изменившиеся курсы</option><option value="true">Полная, без кэша</option></select></label></div>
|
||||
<p className="schedule__hint">Запланированная проверка выполняется на сервере — вкладку можно закрыть. Новые настройки не прерывают уже начатую проверку. Если другая проверка ещё идёт, совпавший запуск пропускается.</p>
|
||||
{schedule.lastAttemptAt&&<p className="schedule__history">{schedule.lastOutcome==='skipped'?'Последний запуск пропущен: предыдущая проверка ещё не завершена.':'Последний автоматический запуск:'} {date(schedule.lastAttemptAt)} · МСК</p>}
|
||||
{schedule.lastError&&<p className="schedule__error" role="status">Ошибка фоновой проверки: {schedule.lastError}</p>}
|
||||
{error&&<p className="schedule__error" role="alert">{error}</p>}
|
||||
<div className="schedule__actions"><button type="submit" className="button button--primary" disabled={busy||!dirty}>{busy&&<LoaderCircle size={15} className="spin"/>}Сохранить расписание</button><span role="status">{saved?<><Check size={15}/>Расписание сохранено</>:dirty?'Есть несохранённые изменения':''}</span></div>
|
||||
</form></details></section>;
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
name: miem-courses
|
||||
services:
|
||||
courses:
|
||||
build: .
|
||||
image: miem-courses:1.3.0
|
||||
restart: unless-stopped
|
||||
init: true
|
||||
user: "1000:1000"
|
||||
ports:
|
||||
- "127.0.0.1:${COURSES_PORT:-3011}:3000"
|
||||
environment:
|
||||
NODE_ENV: production
|
||||
NEXT_TELEMETRY_DISABLED: "1"
|
||||
COURSES_ORIGIN: https://miem.space
|
||||
DATABASE_URL: ${DATABASE_URL:?Set DATABASE_URL for the dedicated courses PostgreSQL database}
|
||||
networks:
|
||||
- database
|
||||
read_only: true
|
||||
tmpfs:
|
||||
- /tmp:size=128m,mode=1777
|
||||
cap_drop:
|
||||
- ALL
|
||||
security_opt:
|
||||
- no-new-privileges:true
|
||||
- seccomp=./deploy/chromium-seccomp.json
|
||||
shm_size: 128m
|
||||
mem_limit: 1g
|
||||
cpus: 1.0
|
||||
networks:
|
||||
database:
|
||||
external: true
|
||||
name: ${COURSES_DATABASE_NETWORK:-miem_workers_default}
|
||||
@@ -0,0 +1,202 @@
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. We also recommend that a
|
||||
file or class name and description of purpose be included on the
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Portions Copyright (c) Microsoft Corporation.
|
||||
Portions Copyright 2017 Google Inc.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
@@ -0,0 +1,8 @@
|
||||
# Chromium sandbox
|
||||
|
||||
chromium-seccomp.json — адаптация профиля Microsoft Playwright v1.64.1:
|
||||
https://github.com/microsoft/playwright/blob/v1.64.1/utils/docker/seccomp_profile.json
|
||||
|
||||
Лицензия Apache-2.0 сохранена в PLAYWRIGHT-LICENSE. Профиль разрешает clone/setns/unshare/chroot для Chromium user namespaces и используется с chromiumSandbox:true, UID1000, no-new-privileges и cap-drop ALL. Не заменять seccomp на unconfined или Chromium sandbox на --no-sandbox. Дополнительно разрешён syscall chroot: он нужен Chromium внутри собственного user namespace; host/container capability SYS_CHROOT не добавляется, kernel permission checks сохраняются. Read-only rootfs и временные /tmp,/dev/shm сохраняются.
|
||||
|
||||
Nginx-сниппет повышает read timeout только для /courses до85s, чтобы browser deadline60s и сохранение данных завершались до90s lease. Другие маршруты и сервисы не меняются.
|
||||
@@ -0,0 +1,832 @@
|
||||
{
|
||||
"defaultAction": "SCMP_ACT_ERRNO",
|
||||
"archMap": [
|
||||
{
|
||||
"architecture": "SCMP_ARCH_X86_64",
|
||||
"subArchitectures": [
|
||||
"SCMP_ARCH_X86",
|
||||
"SCMP_ARCH_X32"
|
||||
]
|
||||
},
|
||||
{
|
||||
"architecture": "SCMP_ARCH_AARCH64",
|
||||
"subArchitectures": [
|
||||
"SCMP_ARCH_ARM"
|
||||
]
|
||||
},
|
||||
{
|
||||
"architecture": "SCMP_ARCH_MIPS64",
|
||||
"subArchitectures": [
|
||||
"SCMP_ARCH_MIPS",
|
||||
"SCMP_ARCH_MIPS64N32"
|
||||
]
|
||||
},
|
||||
{
|
||||
"architecture": "SCMP_ARCH_MIPS64N32",
|
||||
"subArchitectures": [
|
||||
"SCMP_ARCH_MIPS",
|
||||
"SCMP_ARCH_MIPS64"
|
||||
]
|
||||
},
|
||||
{
|
||||
"architecture": "SCMP_ARCH_MIPSEL64",
|
||||
"subArchitectures": [
|
||||
"SCMP_ARCH_MIPSEL",
|
||||
"SCMP_ARCH_MIPSEL64N32"
|
||||
]
|
||||
},
|
||||
{
|
||||
"architecture": "SCMP_ARCH_MIPSEL64N32",
|
||||
"subArchitectures": [
|
||||
"SCMP_ARCH_MIPSEL",
|
||||
"SCMP_ARCH_MIPSEL64"
|
||||
]
|
||||
},
|
||||
{
|
||||
"architecture": "SCMP_ARCH_S390X",
|
||||
"subArchitectures": [
|
||||
"SCMP_ARCH_S390"
|
||||
]
|
||||
}
|
||||
],
|
||||
"syscalls": [
|
||||
{
|
||||
"comment": "Allow Chromium user namespaces and chroot inside its namespace",
|
||||
"names": [
|
||||
"clone",
|
||||
"setns",
|
||||
"unshare",
|
||||
"chroot"
|
||||
],
|
||||
"action": "SCMP_ACT_ALLOW",
|
||||
"args": [],
|
||||
"includes": {},
|
||||
"excludes": {}
|
||||
},
|
||||
{
|
||||
"names": [
|
||||
"accept",
|
||||
"accept4",
|
||||
"access",
|
||||
"adjtimex",
|
||||
"alarm",
|
||||
"bind",
|
||||
"brk",
|
||||
"capget",
|
||||
"capset",
|
||||
"chdir",
|
||||
"chmod",
|
||||
"chown",
|
||||
"chown32",
|
||||
"clock_adjtime",
|
||||
"clock_adjtime64",
|
||||
"clock_getres",
|
||||
"clock_getres_time64",
|
||||
"clock_gettime",
|
||||
"clock_gettime64",
|
||||
"clock_nanosleep",
|
||||
"clock_nanosleep_time64",
|
||||
"close",
|
||||
"connect",
|
||||
"copy_file_range",
|
||||
"creat",
|
||||
"dup",
|
||||
"dup2",
|
||||
"dup3",
|
||||
"epoll_create",
|
||||
"epoll_create1",
|
||||
"epoll_ctl",
|
||||
"epoll_ctl_old",
|
||||
"epoll_pwait",
|
||||
"epoll_wait",
|
||||
"epoll_wait_old",
|
||||
"eventfd",
|
||||
"eventfd2",
|
||||
"execve",
|
||||
"execveat",
|
||||
"exit",
|
||||
"exit_group",
|
||||
"faccessat",
|
||||
"fadvise64",
|
||||
"fadvise64_64",
|
||||
"fallocate",
|
||||
"fanotify_mark",
|
||||
"fchdir",
|
||||
"fchmod",
|
||||
"fchmodat",
|
||||
"fchown",
|
||||
"fchown32",
|
||||
"fchownat",
|
||||
"fcntl",
|
||||
"fcntl64",
|
||||
"fdatasync",
|
||||
"fgetxattr",
|
||||
"flistxattr",
|
||||
"flock",
|
||||
"fork",
|
||||
"fremovexattr",
|
||||
"fsetxattr",
|
||||
"fstat",
|
||||
"fstat64",
|
||||
"fstatat64",
|
||||
"fstatfs",
|
||||
"fstatfs64",
|
||||
"fsync",
|
||||
"ftruncate",
|
||||
"ftruncate64",
|
||||
"futex",
|
||||
"futex_time64",
|
||||
"futimesat",
|
||||
"getcpu",
|
||||
"getcwd",
|
||||
"getdents",
|
||||
"getdents64",
|
||||
"getegid",
|
||||
"getegid32",
|
||||
"geteuid",
|
||||
"geteuid32",
|
||||
"getgid",
|
||||
"getgid32",
|
||||
"getgroups",
|
||||
"getgroups32",
|
||||
"getitimer",
|
||||
"getpeername",
|
||||
"getpgid",
|
||||
"getpgrp",
|
||||
"getpid",
|
||||
"getppid",
|
||||
"getpriority",
|
||||
"getrandom",
|
||||
"getresgid",
|
||||
"getresgid32",
|
||||
"getresuid",
|
||||
"getresuid32",
|
||||
"getrlimit",
|
||||
"get_robust_list",
|
||||
"getrusage",
|
||||
"getsid",
|
||||
"getsockname",
|
||||
"getsockopt",
|
||||
"get_thread_area",
|
||||
"gettid",
|
||||
"gettimeofday",
|
||||
"getuid",
|
||||
"getuid32",
|
||||
"getxattr",
|
||||
"inotify_add_watch",
|
||||
"inotify_init",
|
||||
"inotify_init1",
|
||||
"inotify_rm_watch",
|
||||
"io_cancel",
|
||||
"ioctl",
|
||||
"io_destroy",
|
||||
"io_getevents",
|
||||
"io_pgetevents",
|
||||
"io_pgetevents_time64",
|
||||
"ioprio_get",
|
||||
"ioprio_set",
|
||||
"io_setup",
|
||||
"io_submit",
|
||||
"io_uring_enter",
|
||||
"io_uring_register",
|
||||
"io_uring_setup",
|
||||
"ipc",
|
||||
"kill",
|
||||
"lchown",
|
||||
"lchown32",
|
||||
"lgetxattr",
|
||||
"link",
|
||||
"linkat",
|
||||
"listen",
|
||||
"listxattr",
|
||||
"llistxattr",
|
||||
"_llseek",
|
||||
"lremovexattr",
|
||||
"lseek",
|
||||
"lsetxattr",
|
||||
"lstat",
|
||||
"lstat64",
|
||||
"madvise",
|
||||
"membarrier",
|
||||
"memfd_create",
|
||||
"mincore",
|
||||
"mkdir",
|
||||
"mkdirat",
|
||||
"mknod",
|
||||
"mknodat",
|
||||
"mlock",
|
||||
"mlock2",
|
||||
"mlockall",
|
||||
"mmap",
|
||||
"mmap2",
|
||||
"mprotect",
|
||||
"mq_getsetattr",
|
||||
"mq_notify",
|
||||
"mq_open",
|
||||
"mq_timedreceive",
|
||||
"mq_timedreceive_time64",
|
||||
"mq_timedsend",
|
||||
"mq_timedsend_time64",
|
||||
"mq_unlink",
|
||||
"mremap",
|
||||
"msgctl",
|
||||
"msgget",
|
||||
"msgrcv",
|
||||
"msgsnd",
|
||||
"msync",
|
||||
"munlock",
|
||||
"munlockall",
|
||||
"munmap",
|
||||
"nanosleep",
|
||||
"newfstatat",
|
||||
"_newselect",
|
||||
"open",
|
||||
"openat",
|
||||
"pause",
|
||||
"pipe",
|
||||
"pipe2",
|
||||
"poll",
|
||||
"ppoll",
|
||||
"ppoll_time64",
|
||||
"prctl",
|
||||
"pread64",
|
||||
"preadv",
|
||||
"preadv2",
|
||||
"prlimit64",
|
||||
"pselect6",
|
||||
"pselect6_time64",
|
||||
"pwrite64",
|
||||
"pwritev",
|
||||
"pwritev2",
|
||||
"read",
|
||||
"readahead",
|
||||
"readlink",
|
||||
"readlinkat",
|
||||
"readv",
|
||||
"recv",
|
||||
"recvfrom",
|
||||
"recvmmsg",
|
||||
"recvmmsg_time64",
|
||||
"recvmsg",
|
||||
"remap_file_pages",
|
||||
"removexattr",
|
||||
"rename",
|
||||
"renameat",
|
||||
"renameat2",
|
||||
"restart_syscall",
|
||||
"rmdir",
|
||||
"rseq",
|
||||
"rt_sigaction",
|
||||
"rt_sigpending",
|
||||
"rt_sigprocmask",
|
||||
"rt_sigqueueinfo",
|
||||
"rt_sigreturn",
|
||||
"rt_sigsuspend",
|
||||
"rt_sigtimedwait",
|
||||
"rt_sigtimedwait_time64",
|
||||
"rt_tgsigqueueinfo",
|
||||
"sched_getaffinity",
|
||||
"sched_getattr",
|
||||
"sched_getparam",
|
||||
"sched_get_priority_max",
|
||||
"sched_get_priority_min",
|
||||
"sched_getscheduler",
|
||||
"sched_rr_get_interval",
|
||||
"sched_rr_get_interval_time64",
|
||||
"sched_setaffinity",
|
||||
"sched_setattr",
|
||||
"sched_setparam",
|
||||
"sched_setscheduler",
|
||||
"sched_yield",
|
||||
"seccomp",
|
||||
"select",
|
||||
"semctl",
|
||||
"semget",
|
||||
"semop",
|
||||
"semtimedop",
|
||||
"semtimedop_time64",
|
||||
"send",
|
||||
"sendfile",
|
||||
"sendfile64",
|
||||
"sendmmsg",
|
||||
"sendmsg",
|
||||
"sendto",
|
||||
"setfsgid",
|
||||
"setfsgid32",
|
||||
"setfsuid",
|
||||
"setfsuid32",
|
||||
"setgid",
|
||||
"setgid32",
|
||||
"setgroups",
|
||||
"setgroups32",
|
||||
"setitimer",
|
||||
"setpgid",
|
||||
"setpriority",
|
||||
"setregid",
|
||||
"setregid32",
|
||||
"setresgid",
|
||||
"setresgid32",
|
||||
"setresuid",
|
||||
"setresuid32",
|
||||
"setreuid",
|
||||
"setreuid32",
|
||||
"setrlimit",
|
||||
"set_robust_list",
|
||||
"setsid",
|
||||
"setsockopt",
|
||||
"set_thread_area",
|
||||
"set_tid_address",
|
||||
"setuid",
|
||||
"setuid32",
|
||||
"setxattr",
|
||||
"shmat",
|
||||
"shmctl",
|
||||
"shmdt",
|
||||
"shmget",
|
||||
"shutdown",
|
||||
"sigaltstack",
|
||||
"signalfd",
|
||||
"signalfd4",
|
||||
"sigprocmask",
|
||||
"sigreturn",
|
||||
"socket",
|
||||
"socketcall",
|
||||
"socketpair",
|
||||
"splice",
|
||||
"stat",
|
||||
"stat64",
|
||||
"statfs",
|
||||
"statfs64",
|
||||
"statx",
|
||||
"symlink",
|
||||
"symlinkat",
|
||||
"sync",
|
||||
"sync_file_range",
|
||||
"syncfs",
|
||||
"sysinfo",
|
||||
"tee",
|
||||
"tgkill",
|
||||
"time",
|
||||
"timer_create",
|
||||
"timer_delete",
|
||||
"timer_getoverrun",
|
||||
"timer_gettime",
|
||||
"timer_gettime64",
|
||||
"timer_settime",
|
||||
"timer_settime64",
|
||||
"timerfd_create",
|
||||
"timerfd_gettime",
|
||||
"timerfd_gettime64",
|
||||
"timerfd_settime",
|
||||
"timerfd_settime64",
|
||||
"times",
|
||||
"tkill",
|
||||
"truncate",
|
||||
"truncate64",
|
||||
"ugetrlimit",
|
||||
"umask",
|
||||
"uname",
|
||||
"unlink",
|
||||
"unlinkat",
|
||||
"utime",
|
||||
"utimensat",
|
||||
"utimensat_time64",
|
||||
"utimes",
|
||||
"vfork",
|
||||
"vmsplice",
|
||||
"wait4",
|
||||
"waitid",
|
||||
"waitpid",
|
||||
"write",
|
||||
"writev"
|
||||
],
|
||||
"action": "SCMP_ACT_ALLOW",
|
||||
"args": [],
|
||||
"comment": "",
|
||||
"includes": {},
|
||||
"excludes": {}
|
||||
},
|
||||
{
|
||||
"names": [
|
||||
"ptrace"
|
||||
],
|
||||
"action": "SCMP_ACT_ALLOW",
|
||||
"args": null,
|
||||
"comment": "",
|
||||
"includes": {
|
||||
"minKernel": "4.8"
|
||||
},
|
||||
"excludes": {}
|
||||
},
|
||||
{
|
||||
"names": [
|
||||
"personality"
|
||||
],
|
||||
"action": "SCMP_ACT_ALLOW",
|
||||
"args": [
|
||||
{
|
||||
"index": 0,
|
||||
"value": 0,
|
||||
"valueTwo": 0,
|
||||
"op": "SCMP_CMP_EQ"
|
||||
}
|
||||
],
|
||||
"comment": "",
|
||||
"includes": {},
|
||||
"excludes": {}
|
||||
},
|
||||
{
|
||||
"names": [
|
||||
"personality"
|
||||
],
|
||||
"action": "SCMP_ACT_ALLOW",
|
||||
"args": [
|
||||
{
|
||||
"index": 0,
|
||||
"value": 8,
|
||||
"valueTwo": 0,
|
||||
"op": "SCMP_CMP_EQ"
|
||||
}
|
||||
],
|
||||
"comment": "",
|
||||
"includes": {},
|
||||
"excludes": {}
|
||||
},
|
||||
{
|
||||
"names": [
|
||||
"personality"
|
||||
],
|
||||
"action": "SCMP_ACT_ALLOW",
|
||||
"args": [
|
||||
{
|
||||
"index": 0,
|
||||
"value": 131072,
|
||||
"valueTwo": 0,
|
||||
"op": "SCMP_CMP_EQ"
|
||||
}
|
||||
],
|
||||
"comment": "",
|
||||
"includes": {},
|
||||
"excludes": {}
|
||||
},
|
||||
{
|
||||
"names": [
|
||||
"personality"
|
||||
],
|
||||
"action": "SCMP_ACT_ALLOW",
|
||||
"args": [
|
||||
{
|
||||
"index": 0,
|
||||
"value": 131080,
|
||||
"valueTwo": 0,
|
||||
"op": "SCMP_CMP_EQ"
|
||||
}
|
||||
],
|
||||
"comment": "",
|
||||
"includes": {},
|
||||
"excludes": {}
|
||||
},
|
||||
{
|
||||
"names": [
|
||||
"personality"
|
||||
],
|
||||
"action": "SCMP_ACT_ALLOW",
|
||||
"args": [
|
||||
{
|
||||
"index": 0,
|
||||
"value": 4294967295,
|
||||
"valueTwo": 0,
|
||||
"op": "SCMP_CMP_EQ"
|
||||
}
|
||||
],
|
||||
"comment": "",
|
||||
"includes": {},
|
||||
"excludes": {}
|
||||
},
|
||||
{
|
||||
"names": [
|
||||
"sync_file_range2"
|
||||
],
|
||||
"action": "SCMP_ACT_ALLOW",
|
||||
"args": [],
|
||||
"comment": "",
|
||||
"includes": {
|
||||
"arches": [
|
||||
"ppc64le"
|
||||
]
|
||||
},
|
||||
"excludes": {}
|
||||
},
|
||||
{
|
||||
"names": [
|
||||
"arm_fadvise64_64",
|
||||
"arm_sync_file_range",
|
||||
"sync_file_range2",
|
||||
"breakpoint",
|
||||
"cacheflush",
|
||||
"set_tls"
|
||||
],
|
||||
"action": "SCMP_ACT_ALLOW",
|
||||
"args": [],
|
||||
"comment": "",
|
||||
"includes": {
|
||||
"arches": [
|
||||
"arm",
|
||||
"arm64"
|
||||
]
|
||||
},
|
||||
"excludes": {}
|
||||
},
|
||||
{
|
||||
"names": [
|
||||
"arch_prctl"
|
||||
],
|
||||
"action": "SCMP_ACT_ALLOW",
|
||||
"args": [],
|
||||
"comment": "",
|
||||
"includes": {
|
||||
"arches": [
|
||||
"amd64",
|
||||
"x32"
|
||||
]
|
||||
},
|
||||
"excludes": {}
|
||||
},
|
||||
{
|
||||
"names": [
|
||||
"modify_ldt"
|
||||
],
|
||||
"action": "SCMP_ACT_ALLOW",
|
||||
"args": [],
|
||||
"comment": "",
|
||||
"includes": {
|
||||
"arches": [
|
||||
"amd64",
|
||||
"x32",
|
||||
"x86"
|
||||
]
|
||||
},
|
||||
"excludes": {}
|
||||
},
|
||||
{
|
||||
"names": [
|
||||
"s390_pci_mmio_read",
|
||||
"s390_pci_mmio_write",
|
||||
"s390_runtime_instr"
|
||||
],
|
||||
"action": "SCMP_ACT_ALLOW",
|
||||
"args": [],
|
||||
"comment": "",
|
||||
"includes": {
|
||||
"arches": [
|
||||
"s390",
|
||||
"s390x"
|
||||
]
|
||||
},
|
||||
"excludes": {}
|
||||
},
|
||||
{
|
||||
"names": [
|
||||
"open_by_handle_at"
|
||||
],
|
||||
"action": "SCMP_ACT_ALLOW",
|
||||
"args": [],
|
||||
"comment": "",
|
||||
"includes": {
|
||||
"caps": [
|
||||
"CAP_DAC_READ_SEARCH"
|
||||
]
|
||||
},
|
||||
"excludes": {}
|
||||
},
|
||||
{
|
||||
"names": [
|
||||
"bpf",
|
||||
"clone",
|
||||
"fanotify_init",
|
||||
"lookup_dcookie",
|
||||
"mount",
|
||||
"name_to_handle_at",
|
||||
"perf_event_open",
|
||||
"quotactl",
|
||||
"setdomainname",
|
||||
"sethostname",
|
||||
"setns",
|
||||
"syslog",
|
||||
"umount",
|
||||
"umount2",
|
||||
"unshare"
|
||||
],
|
||||
"action": "SCMP_ACT_ALLOW",
|
||||
"args": [],
|
||||
"comment": "",
|
||||
"includes": {
|
||||
"caps": [
|
||||
"CAP_SYS_ADMIN"
|
||||
]
|
||||
},
|
||||
"excludes": {}
|
||||
},
|
||||
{
|
||||
"names": [
|
||||
"clone"
|
||||
],
|
||||
"action": "SCMP_ACT_ALLOW",
|
||||
"args": [
|
||||
{
|
||||
"index": 0,
|
||||
"value": 2114060288,
|
||||
"valueTwo": 0,
|
||||
"op": "SCMP_CMP_MASKED_EQ"
|
||||
}
|
||||
],
|
||||
"comment": "",
|
||||
"includes": {},
|
||||
"excludes": {
|
||||
"caps": [
|
||||
"CAP_SYS_ADMIN"
|
||||
],
|
||||
"arches": [
|
||||
"s390",
|
||||
"s390x"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"names": [
|
||||
"clone"
|
||||
],
|
||||
"action": "SCMP_ACT_ALLOW",
|
||||
"args": [
|
||||
{
|
||||
"index": 1,
|
||||
"value": 2114060288,
|
||||
"valueTwo": 0,
|
||||
"op": "SCMP_CMP_MASKED_EQ"
|
||||
}
|
||||
],
|
||||
"comment": "s390 parameter ordering for clone is different",
|
||||
"includes": {
|
||||
"arches": [
|
||||
"s390",
|
||||
"s390x"
|
||||
]
|
||||
},
|
||||
"excludes": {
|
||||
"caps": [
|
||||
"CAP_SYS_ADMIN"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"names": [
|
||||
"reboot"
|
||||
],
|
||||
"action": "SCMP_ACT_ALLOW",
|
||||
"args": [],
|
||||
"comment": "",
|
||||
"includes": {
|
||||
"caps": [
|
||||
"CAP_SYS_BOOT"
|
||||
]
|
||||
},
|
||||
"excludes": {}
|
||||
},
|
||||
{
|
||||
"names": [
|
||||
"chroot"
|
||||
],
|
||||
"action": "SCMP_ACT_ALLOW",
|
||||
"args": [],
|
||||
"comment": "",
|
||||
"includes": {
|
||||
"caps": [
|
||||
"CAP_SYS_CHROOT"
|
||||
]
|
||||
},
|
||||
"excludes": {}
|
||||
},
|
||||
{
|
||||
"names": [
|
||||
"delete_module",
|
||||
"init_module",
|
||||
"finit_module"
|
||||
],
|
||||
"action": "SCMP_ACT_ALLOW",
|
||||
"args": [],
|
||||
"comment": "",
|
||||
"includes": {
|
||||
"caps": [
|
||||
"CAP_SYS_MODULE"
|
||||
]
|
||||
},
|
||||
"excludes": {}
|
||||
},
|
||||
{
|
||||
"names": [
|
||||
"acct"
|
||||
],
|
||||
"action": "SCMP_ACT_ALLOW",
|
||||
"args": [],
|
||||
"comment": "",
|
||||
"includes": {
|
||||
"caps": [
|
||||
"CAP_SYS_PACCT"
|
||||
]
|
||||
},
|
||||
"excludes": {}
|
||||
},
|
||||
{
|
||||
"names": [
|
||||
"kcmp",
|
||||
"process_vm_readv",
|
||||
"process_vm_writev",
|
||||
"ptrace"
|
||||
],
|
||||
"action": "SCMP_ACT_ALLOW",
|
||||
"args": [],
|
||||
"comment": "",
|
||||
"includes": {
|
||||
"caps": [
|
||||
"CAP_SYS_PTRACE"
|
||||
]
|
||||
},
|
||||
"excludes": {}
|
||||
},
|
||||
{
|
||||
"names": [
|
||||
"iopl",
|
||||
"ioperm"
|
||||
],
|
||||
"action": "SCMP_ACT_ALLOW",
|
||||
"args": [],
|
||||
"comment": "",
|
||||
"includes": {
|
||||
"caps": [
|
||||
"CAP_SYS_RAWIO"
|
||||
]
|
||||
},
|
||||
"excludes": {}
|
||||
},
|
||||
{
|
||||
"names": [
|
||||
"settimeofday",
|
||||
"stime",
|
||||
"clock_settime"
|
||||
],
|
||||
"action": "SCMP_ACT_ALLOW",
|
||||
"args": [],
|
||||
"comment": "",
|
||||
"includes": {
|
||||
"caps": [
|
||||
"CAP_SYS_TIME"
|
||||
]
|
||||
},
|
||||
"excludes": {}
|
||||
},
|
||||
{
|
||||
"names": [
|
||||
"vhangup"
|
||||
],
|
||||
"action": "SCMP_ACT_ALLOW",
|
||||
"args": [],
|
||||
"comment": "",
|
||||
"includes": {
|
||||
"caps": [
|
||||
"CAP_SYS_TTY_CONFIG"
|
||||
]
|
||||
},
|
||||
"excludes": {}
|
||||
},
|
||||
{
|
||||
"names": [
|
||||
"get_mempolicy",
|
||||
"mbind",
|
||||
"set_mempolicy"
|
||||
],
|
||||
"action": "SCMP_ACT_ALLOW",
|
||||
"args": [],
|
||||
"comment": "",
|
||||
"includes": {
|
||||
"caps": [
|
||||
"CAP_SYS_NICE"
|
||||
]
|
||||
},
|
||||
"excludes": {}
|
||||
},
|
||||
{
|
||||
"names": [
|
||||
"syslog"
|
||||
],
|
||||
"action": "SCMP_ACT_ALLOW",
|
||||
"args": [],
|
||||
"comment": "",
|
||||
"includes": {
|
||||
"caps": [
|
||||
"CAP_SYSLOG"
|
||||
]
|
||||
},
|
||||
"excludes": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
# Include inside the existing TLS server for miem.space; preserve the /courses prefix.
|
||||
location = /courses {
|
||||
proxy_pass http://127.0.0.1:3011;
|
||||
include /etc/nginx/proxy_params;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header X-Forwarded-Host $host;
|
||||
proxy_read_timeout 85s;
|
||||
}
|
||||
location ^~ /courses/ {
|
||||
proxy_pass http://127.0.0.1:3011;
|
||||
include /etc/nginx/proxy_params;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header X-Forwarded-Host $host;
|
||||
proxy_read_timeout 85s;
|
||||
}
|
||||
@@ -0,0 +1,69 @@
|
||||
import {parseHTML} from 'linkedom';
|
||||
import type {Check} from './model.ts';
|
||||
|
||||
export const RULES_VERSION='access-1';
|
||||
export const CHECK_TTL=24*60*60*1000;
|
||||
export function checkCurrent(check:Check|undefined,now=Date.now()){
|
||||
return !!check&&check.rulesVersion===RULES_VERSION&&check.profile==='anonymous'&&now-Date.parse(check.checkedAt)<(['unknown','blocked','network','timeout'].includes(check.status)?600000:CHECK_TTL);
|
||||
}
|
||||
export function evidenceUrl(raw:string){const u=new URL(raw);for(const key of [...u.searchParams.keys()])if(!['pid','id','bcode'].includes(key))u.searchParams.delete(key);u.hash='';return u.href;}
|
||||
export function result(url:string,status:Check['status'],reason:string,extra:Partial<Check>={}):Check{
|
||||
return {url,status,reason,checkedAt:new Date().toISOString(),rulesVersion:RULES_VERSION,profile:'anonymous',...extra};
|
||||
}
|
||||
function evidence(url:string,text:string,rule:string){const normalized=text.replace(/\s+/g,' ').trim(),at=normalized.search(/публикация\s+изъята|для\s+(?:получения\s+)?(?:доступа|чтения|просмотра)|чтобы\s+получить\s+полный\s+доступ|sign in to|чтение\s+книги\s+недоступно|книга\s+недоступна/i);return {text:normalized.slice(Math.max(0,at),Math.max(0,at)+500),url:evidenceUrl(url),rule};}
|
||||
function removeHidden(document:ReturnType<typeof parseHTML>['document']){
|
||||
document.querySelectorAll('script,style,[hidden],[aria-hidden="true"],.hidden').forEach(n=>n.remove());
|
||||
document.querySelectorAll('[style]').forEach(n=>{if(/(?:^|;)\s*(?:display\s*:\s*none|visibility\s*:\s*(?:hidden|collapse))\s*(?:!important)?\s*(?:;|$)/i.test(n.getAttribute('style')??''))n.remove();});
|
||||
}
|
||||
export function accessMessage(url:string,text:string):Check|null{
|
||||
const normalized=text.replace(/\s+/g,' ').trim();
|
||||
const preview=/гостев[а-я]*\s+режим[а-я]*[^.]{0,180}?(\d{1,4})\s*страниц/i.exec(normalized)??/доступн[а-я]*\s+(?:только\s+|первые\s+)?(\d{1,4})\s+страниц/i.exec(normalized);
|
||||
if(preview&&/гостев|ознакомительн|фрагмент/i.test(normalized))return result(url,'restricted',`Доступен только ознакомительный фрагмент — до ${preview[1]} страниц; полный текст недоступен.`,{evidence:evidence(url,normalized.slice(Math.max(0,preview.index-150),preview.index+200),'reader-preview-pages')});
|
||||
if(/публикация\s+изъята\s+из\s+фонда/i.test(normalized))return result(url,'broken','Публикация изъята из фонда.',{evidence:evidence(url,normalized,'publication-withdrawn')});
|
||||
if(/\/bookshelf\/\d+\/reading/.test(new URL(url).pathname)&&/для продолжения работы с сайтом.{0,60}войдите в систему/i.test(normalized))return result(url,'auth','Для чтения книги требуется авторизация.',{evidence:evidence(url,normalized,'ibooks-reader-login')});
|
||||
const subscription=/(?:для\s+(?:полного\s+)?(?:доступа|чтения)|чтобы\s+(?:получить\s+)?полный\s+доступ|полный\s+текст)[^.]{0,250}(?:подписк|оплат|купить)|доступ\s+(?:к\s+полному\s+тексту\s+)?по\s+подписке|читать\s+по\s+подписке|sign in to (?:read|access)[^.]{0,100}subscri/i.test(normalized);
|
||||
const login=/(?:для\s+(?:получения\s+)?(?:доступа\s+к\s+)?(?:чтения|просмотра|доступа)|чтобы\s+(?:получить\s+)?полный\s+доступ)[^.]{0,220}(?:авториз|войти|войдите|вход)|(?:авторизуйтесь|войдите)[^.]{0,100}(?:чтения|полного\s+доступа|чита)|sign in to (?:read|access|view)/i.test(normalized);
|
||||
if(login||subscription)return result(url,subscription?'restricted':'auth',login&&subscription?'Для чтения книги требуются авторизация и подписка.':login?'Для чтения книги требуется авторизация.':'Для чтения книги требуется подписка или оплата.',{evidence:evidence(url,normalized,login&&subscription?'reader-login-subscription':login?'reader-login':'reader-subscription')});
|
||||
if(/(?:доступен|доступно)\s+только\s+(?:ознакомительный\s+)?фрагмент|только\s+ознакомительный\s+фрагмент/i.test(normalized))return result(url,'restricted','Доступен только ознакомительный фрагмент; полный текст недоступен.',{evidence:evidence(url,normalized,'reader-preview')});
|
||||
if(/книга\s+недоступна\s+для\s+чтения|чтение\s+книги\s+недоступно/i.test(normalized))return result(url,'restricted','Чтение книги недоступно. Сайт не указал подтверждённую причину.',{evidence:evidence(url,normalized,'reader-denied')});
|
||||
return null;
|
||||
}
|
||||
export function purchaseOffer(url:string,html:string):Check|null{
|
||||
const {document}=parseHTML(html),host=new URL(url).hostname;
|
||||
removeHidden(document);
|
||||
if(host==='znanium.ru'||host.endsWith('.znanium.ru')){
|
||||
const id=new URL(url).searchParams.get('id');
|
||||
for(const item of document.querySelectorAll('.book-offers__item')){
|
||||
const text=item.textContent??'',control=item.querySelector('.book-offers__ctrl');
|
||||
if(/доступ\s+онлайн/i.test(text)&&/[1-9]\d*(?:[.,]\d+)?\s*₽/.test(text)&&/в корзину/i.test(control?.textContent??'')&&(!id||control?.getAttribute('data-id')===id))return result(url,'restricted','Для чтения книги требуется покупка онлайн-доступа или подписка.',{evidence:evidence(url,text,'znanium-online-purchase')});
|
||||
}
|
||||
}
|
||||
if(/(^|\.)(springer\.com|springernature\.com)$/.test(host)){
|
||||
for(const access of document.querySelectorAll('.app-article-access')){
|
||||
if(/this is an open access book|open access/i.test(access.textContent??''))continue;
|
||||
for(const option of access.querySelectorAll('.buying-option')){
|
||||
if(/ebook/i.test(option.textContent??'')&&/[1-9]/.test(option.querySelector('.price-amount')?.textContent??'')&&/buy\s+ebook/i.test(option.querySelector('button')?.textContent??''))return result(url,'restricted','Для чтения требуется покупка электронной книги или доступ по подписке организации.',{evidence:evidence(url,'Access this book. '+(option.textContent??'').replace(/\s+/g,' '),'springer-ebook-purchase')});
|
||||
}
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
export function classifyResponse(url:string,r:{code:number;url:string;body:string;type:string}):Check{
|
||||
const base={finalUrl:evidenceUrl(r.url),code:r.code};
|
||||
const {document}=parseHTML(/html|text\//i.test(r.type)?r.body:'');
|
||||
removeHidden(document);document.querySelectorAll('nav,footer,header').forEach(n=>n.remove());
|
||||
const title=(document.querySelector('title')?.textContent??'').trim();
|
||||
const text=(document.querySelector('main,[role="main"]')??document.querySelector('body')??document.documentElement)?.textContent??'';
|
||||
const message=accessMessage(r.url,text);if(message?.status==='broken')return {...message,url,...base};
|
||||
if(r.code===404||r.code===410)return result(url,'broken',r.code===404?'Страница материала не найдена.':'Материал удалён.',base);
|
||||
if(r.code===401||/(?:\/|\.)(?:login|signin|sign-in|sso|auth)(?:[/.?]|$)/i.test(r.url))return result(url,'auth','Для чтения материала требуется авторизация.',base);
|
||||
if(r.code===403&&message)return {...message,url,...base};
|
||||
if(r.code===403||r.code===429||/captcha|verify you are human|checking your browser|just a moment|докажите, что вы|проверка браузера/i.test(title+' '+text.slice(0,2000)))return result(url,'blocked','Сайт ограничил автоматическую проверку. Доступ к полному тексту не определён.',base);
|
||||
if(r.code>=400)return result(url,'network',`Сайт временно не предоставил материал (HTTP ${r.code}).`,base);
|
||||
if(r.code<200||r.code>=300)return result(url,'unknown',`Ответ HTTP ${r.code} не подтвердил доступ к материалу.`,base);
|
||||
if(message)return {...message,url,...base};
|
||||
const purchase=purchaseOffer(r.url,r.body);if(purchase)return {...purchase,url,...base};
|
||||
if(/^(404|страница не найдена|page not found|not found)/i.test(title))return result(url,'broken','Сайт показывает страницу «не найдено».',{...base,evidence:evidence(r.url,title,'page-not-found')});
|
||||
if(/application\/pdf/i.test(r.type)&&r.body.startsWith('%PDF-'))return result(url,'unknown','PDF получен без авторизации, но не установлено, содержит ли он полный текст или только фрагмент.',base);
|
||||
return result(url,'unknown','Страница открывается, но доступ к полному тексту не подтверждён.',base);
|
||||
}
|
||||
+131
@@ -0,0 +1,131 @@
|
||||
import {chromium,type Route} from 'playwright';
|
||||
import {requestPublic,safeUrl} from './http.ts';
|
||||
import {waitForHost,limitHost,HostCooldown} from './throttle.ts';
|
||||
import {accessMessage,purchaseOffer,classifyResponse,result,evidenceUrl} from './access.ts';
|
||||
import type {Check} from './model.ts';
|
||||
|
||||
import {provider} from './provider.ts';
|
||||
export function browserRequestAllowed(raw:string,root:string,method:string,resource:string){
|
||||
const url=safeUrl(raw);if(!url)return false;const u=new URL(url);
|
||||
return u.protocol==='https:'&&provider(u.href)===root&&['GET','HEAD','OPTIONS'].includes(method)&&!['image','media','font','manifest'].includes(resource);
|
||||
}
|
||||
export async function checkBrowser(url:string,parentSignal?:AbortSignal):Promise<Check>{
|
||||
const root=provider(url);if(!root)return result(url,'unknown','Для этого сайта нет правил проверки полного текста.');
|
||||
const controller=new AbortController(),timer=setTimeout(()=>controller.abort(),30000);
|
||||
const abort=()=>controller.abort();parentSignal?.addEventListener('abort',abort,{once:true});if(parentSignal?.aborted)controller.abort();
|
||||
let browser:Awaited<ReturnType<typeof chromium.launch>>|undefined;
|
||||
let requests=0,totalBytes=0,failed=false,navigations=0,mainCode=200,redirectTarget:string|null=null,limit:Check|null=null;
|
||||
const active=new Set<Promise<void>>();
|
||||
const slots:Promise<void>[]=Array.from({length:10},()=>Promise.resolve());
|
||||
try{
|
||||
controller.signal.throwIfAborted();browser=await chromium.launch({headless:true,chromiumSandbox:true,timeout:10000,env:{PATH:process.env.PATH??'/usr/bin:/bin',HOME:'/tmp',TMPDIR:'/tmp',LANG:'C.UTF-8'},args:[
|
||||
'--proxy-server=http://127.0.0.1:9','--proxy-bypass-list=<-loopback>','--disable-quic','--force-webrtc-ip-handling-policy=disable_non_proxied_udp',
|
||||
]});
|
||||
controller.signal.throwIfAborted();const context=await browser.newContext({serviceWorkers:'block',acceptDownloads:false,javaScriptEnabled:!['znanium.ru','ibooks.ru','springer.com'].includes(root),viewport:{width:1280,height:900}});
|
||||
controller.signal.addEventListener('abort',()=>{void context.close().catch(()=>{});},{once:true});
|
||||
await context.routeWebSocket('**/*',socket=>socket.close());
|
||||
const handle=async(route:Route)=>{
|
||||
const request=route.request(),target=request.url(),navigation=request.isNavigationRequest();
|
||||
try{
|
||||
if(controller.signal.aborted||limit||!browserRequestAllowed(target,root,request.method(),request.resourceType())){await route.abort();return;}
|
||||
if(totalBytes>32000000)throw new Error('Превышен лимит ресурсов страницы');
|
||||
if(navigation){if(++navigations>8)throw new Error('Слишком много переходов');await waitForHost(new URL(target).hostname,controller.signal);}
|
||||
const allHeaders=await request.allHeaders(),headers:Record<string,string>={};for(const key of ['accept','accept-language','origin','referer','cookie','content-type','user-agent']){const value=allHeaders[key];if(value)headers[key]=value;}
|
||||
const response=await requestPublic(target,{signal:controller.signal,maxBytes:8000000,method:request.method() as 'GET'|'HEAD'|'OPTIONS',headers});
|
||||
totalBytes+=response.body.length;if(totalBytes>32000000)throw new Error('Превышен лимит ресурсов страницы');
|
||||
if([429,503].includes(response.code)){
|
||||
await limitHost(new URL(target).hostname,response.code,response.headers['retry-after']);
|
||||
limit=result(url,'blocked',`Сайт остановил автоматическую проверку (HTTP ${response.code}). Доступ к полному тексту не определён.`);
|
||||
}
|
||||
if(navigation&&request.frame()===page.mainFrame())mainCode=response.code;
|
||||
const fulfilled:Record<string,string>={};
|
||||
for(const [key,value] of Object.entries(response.headers))if(value!==undefined&&!['content-encoding','content-length','transfer-encoding','connection'].includes(key))fulfilled[key]=Array.isArray(value)?value.join('\n'):String(value);
|
||||
if([301,302,303,307,308].includes(response.code)&&response.headers.location){
|
||||
if(!navigation||request.frame()!==page.mainFrame())throw new Error('Перенаправление ресурса не подтверждено');
|
||||
const next=safeUrl(response.headers.location,target);if(!next||!browserRequestAllowed(next,root,'GET','document'))throw new Error('Перенаправление не разрешено');
|
||||
redirectTarget=next;delete fulfilled.location;delete fulfilled['content-type'];
|
||||
// Chromium routing may skip native redirect hops. Start each hop explicitly through the pinned boundary.
|
||||
await route.fulfill({status:200,headers:fulfilled,contentType:'text/html',body:'<!doctype html><html><body></body></html>'});return;
|
||||
}
|
||||
await route.fulfill({status:response.code,headers:fulfilled,body:response.body});
|
||||
}catch(error){
|
||||
if(error instanceof HostCooldown)limit=result(url,'blocked',error.message);
|
||||
else if(navigation||['script','xhr','fetch','stylesheet'].includes(request.resourceType()))failed=true;
|
||||
await route.abort().catch(()=>{});
|
||||
}
|
||||
};
|
||||
// Every request is fulfilled by the pinned Node transport. The dead proxy also prevents direct browser egress.
|
||||
await context.route('**/*',route=>{
|
||||
const request=route.request();
|
||||
if(controller.signal.aborted||limit||!browserRequestAllowed(request.url(),root,request.method(),request.resourceType()))return route.abort().catch(()=>{});
|
||||
if(++requests>100){failed=true;return route.abort().catch(()=>{});}
|
||||
const index=(requests-1)%slots.length,task=slots[index].then(()=>handle(route));slots[index]=task.catch(()=>{});active.add(task);void task.finally(()=>active.delete(task));return task;
|
||||
});
|
||||
const page=await context.newPage();
|
||||
context.on('page',other=>{if(other!==page)void other.close().catch(()=>{});});
|
||||
page.setDefaultTimeout(5000);page.setDefaultNavigationTimeout(20000);
|
||||
const navigate=async(raw:string)=>{let next:string=raw;for(let hop=0;hop<5;hop++){redirectTarget=null;await page.goto(next,{waitUntil:'domcontentloaded'});if(!redirectTarget)return;next=redirectTarget;}throw new Error('Слишком много перенаправлений');};
|
||||
const capture=async()=>{
|
||||
await page.locator('body').waitFor({state:'attached'});
|
||||
// Only rendered text is evidence; hidden login templates and site chrome cannot decide book access.
|
||||
const text=await page.locator('body').evaluate(body=>{
|
||||
const walker=document.createTreeWalker(body,NodeFilter.SHOW_TEXT),parts:string[]=[];let node:Node|null,size=0;
|
||||
while((node=walker.nextNode())&&size<100000){const el=node.parentElement;if(!el||el.closest('header,footer,nav,script,style,[role="navigation"],[role="banner"],[role="contentinfo"]'))continue;const style=getComputedStyle(el);if(style.visibility==='hidden'||style.display==='none'||!el.getClientRects().length)continue;const text=node.textContent??'';parts.push(text);size+=text.length;}
|
||||
return parts.join(' ');
|
||||
});
|
||||
const offers=await page.locator('.book-offers__item,.app-article-access').evaluateAll(nodes=>nodes.filter(node=>{const style=getComputedStyle(node);return style.display!=='none'&&style.visibility!=='hidden'&&node.getClientRects().length>0;}).map(node=>{
|
||||
const clone=node.cloneNode(true) as Element,original=[...node.querySelectorAll('*')],copied=[...clone.querySelectorAll('*')];
|
||||
original.forEach((el,i)=>{const style=getComputedStyle(el);if(style.display==='none'||style.visibility==='hidden'||!el.getClientRects().length)copied[i].remove();});
|
||||
return clone.outerHTML;
|
||||
}).join(''));
|
||||
return {text,offers,url:page.url(),title:await page.title()};
|
||||
};
|
||||
const settle=async()=>{
|
||||
await page.waitForFunction(()=>{
|
||||
const text=document.body?.innerText??'';
|
||||
return /чтение книги недоступно|необходимо авторизоваться|публикация изъята|чтобы получить полный доступ|книга недоступна для чтения|для продолжения работы с сайтом|библиографическое описание|not found|captcha|проверка браузера/i.test(text)||!!document.querySelector('a[href*="/course-viewer/"],a[href*="/viewer/"],a[href*="reader.new.book.ru"],.book-offers__item,.app-article-access');
|
||||
},undefined,{timeout:12000}).catch(()=>{});
|
||||
};
|
||||
await navigate(url);await settle();if(['urait.ru','book.ru'].includes(root))await page.waitForLoadState('networkidle',{timeout:4000}).catch(()=>{});let snapshot=await capture();
|
||||
if(root==='book.ru'){
|
||||
// CSS tooltips become part of the accessible name on hover; retain a stable DOM-text locator.
|
||||
const preview=page.locator('a[href*="reader.new.book.ru"]').filter({hasText:'Ознакомиться',visible:true}).first();
|
||||
if(await preview.count()){
|
||||
await preview.hover({timeout:3000}).catch(()=>{});
|
||||
await page.waitForFunction(()=>/гостевом\s+режиме|доступны\s+только\s+\d+\s+страниц/i.test(document.body?.innerText??''),undefined,{timeout:3000}).catch(()=>{});snapshot=await capture();
|
||||
const tooltip=await preview.evaluate(el=>[el,el.parentElement].filter((node):node is HTMLElement=>!!node).flatMap(node=>['::before','::after'].map(pseudo=>{const style=getComputedStyle(node,pseudo);return style.display!=='none'&&style.visibility!=='hidden'&&style.opacity!=='0'&&!['none','normal'].includes(style.content)?style.content.replace(/^["']|["']$/g,''):'';})).join(' ')).catch(()=>'');
|
||||
snapshot.text+=' '+tooltip;
|
||||
const message=accessMessage(snapshot.url,snapshot.text);if(message)return {...message,url,code:mainCode,finalUrl:evidenceUrl(snapshot.url)};
|
||||
// BOOK.ru distinguishes its locked orange preview action from a full reading action.
|
||||
const lockedPreview=await preview.evaluate(el=>el.className.includes('book_mode_orange__')&&!![...el.querySelectorAll('[class*="book_marker2__"]')].find(node=>{const style=getComputedStyle(node);return style.display!=='none'&&style.visibility!=='hidden'&&node.getClientRects().length>0;}));
|
||||
const fullRead=await page.locator('a[href*="reader.new.book.ru"]').filter({hasText:/^\s*Читать(?: полностью)?\s*$/i}).count();
|
||||
if(lockedPreview&&!fullRead)return result(url,'restricted','Доступен только ознакомительный режим; полный текст книги недоступен.',{code:mainCode,finalUrl:evidenceUrl(snapshot.url),evidence:{url:evidenceUrl(snapshot.url),text:'Ознакомиться',rule:'bookru-locked-preview'}});
|
||||
|
||||
}
|
||||
}
|
||||
const known=accessMessage(snapshot.url,snapshot.text)??purchaseOffer(snapshot.url,snapshot.offers);
|
||||
if(known)return {...known,url,code:mainCode,finalUrl:evidenceUrl(snapshot.url)};
|
||||
if(root==='urait.ru'||root==='ibooks.ru'){
|
||||
// Follow only a book reader link already present on this exact card, never subscription/purchase controls.
|
||||
const modern=root==='urait.ru'?await page.locator('a[href*="/course-viewer/"]').first().getAttribute('href').catch(()=>null):null;
|
||||
const href=root==='urait.ru'?modern??await page.locator('a[href*="/viewer/"]').first().getAttribute('href').catch(()=>null):snapshot.text.match(/https:\/\/ibooks\.ru\/bookshelf\/\d+\/reading/)?.[0]??null;
|
||||
const reader=href?safeUrl(href,snapshot.url):null;
|
||||
if(reader&&provider(reader)===root){await navigate(reader);await settle();snapshot=await capture();
|
||||
if(root==='urait.ru'&&new URL(snapshot.url).pathname.startsWith('/viewer/')){
|
||||
const nextHref=await page.locator('a[href*="/course-viewer/"]').first().getAttribute('href').catch(()=>null),next=nextHref?safeUrl(nextHref,snapshot.url):null;
|
||||
if(next&&provider(next)===root){await navigate(next);await settle();snapshot=await capture();}
|
||||
}
|
||||
const message=accessMessage(snapshot.url,snapshot.text);if(message)return {...message,url,code:mainCode,finalUrl:evidenceUrl(snapshot.url)};
|
||||
}
|
||||
}
|
||||
if(limit)return limit;
|
||||
const escape=(text:string)=>text.replaceAll('&','&').replaceAll('<','<');
|
||||
const classified=classifyResponse(url,{code:mainCode,url:snapshot.url,body:'<html><title>'+escape(snapshot.title)+'</title><body>'+escape(snapshot.text)+'</body></html>',type:'text/html'});
|
||||
if(mainCode===403&&classified.status==='blocked')await limitHost(new URL(snapshot.url).hostname,403);
|
||||
if(failed&&classified.status==='unknown')classified.reason='Страница загрузилась не полностью. Доступ к полному тексту не подтверждён.';
|
||||
return classified;
|
||||
}catch(error){
|
||||
if(limit)return limit;
|
||||
return result(url,'unknown',controller.signal.aborted?'Не удалось определить доступ к полному тексту за отведённое время.':'Не удалось завершить браузерную проверку. Доступ к полному тексту не подтверждён.');
|
||||
}finally{clearTimeout(timer);parentSignal?.removeEventListener('abort',abort);controller.abort();await browser?.close().catch(()=>{});await Promise.allSettled(active);}
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
import {parseHTML} from 'linkedom';
|
||||
import type {Check,Reference} from './model';
|
||||
import {HostCooldown,limitHost} from './throttle.ts';
|
||||
import {safeUrl,retrieve} from './http.ts';
|
||||
import {classifyResponse,result} from './access.ts';
|
||||
import {provider} from './provider.ts';
|
||||
export function clean(text:string){return text.replace(/\s+/g,' ').trim();}
|
||||
export function trimUrl(value:string){let v=value.replace(/[.,;:!?]+$/g,'');for(const [open,close] of [['(',')'],['[',']'],['{','}']])while(v.endsWith(close)&&v.split(close).length>v.split(open).length)v=v.slice(0,-1);return v;}
|
||||
export function parseCatalog(html:string,url:string,root:string){
|
||||
const {document}=parseHTML(html),courses=new Map<string,{url:string;name:string}>(),pages=new Set<string>();
|
||||
for(const a of document.querySelectorAll('a[href]')){const href=safeUrl(a.getAttribute('href')!,url);if(!href)continue;
|
||||
const u=new URL(href),r=new URL(root);if(u.origin!==r.origin)continue;
|
||||
if(new RegExp('^'+r.pathname+'/[0-9]+\\.html$').test(u.pathname))courses.set(u.origin+u.pathname,{url:u.origin+u.pathname,name:clean(a.textContent??'')});
|
||||
if(new RegExp('^'+r.pathname+'/page[0-9]+\\.html$').test(u.pathname)&&(url===root||u.search===new URL(url).search))pages.add(href);}
|
||||
return {courses:[...courses.values()],pages:[...pages]};}
|
||||
export function parseCourse(html:string,url:string){
|
||||
const {document}=parseHTML(html),refs=new Map<string,Reference>();
|
||||
const sections=[...document.querySelectorAll('.pud__section')].filter(s=>/список литературы|основная литература|дополнительная литература|reading list|bibliography/i.test(s.querySelector('h4,h3,h2')?.textContent??''));
|
||||
let found=sections.length>0;
|
||||
if(!found){const h=[...document.querySelectorAll('h2,h3,h4')].find(h=>/^(список литературы|литература|reading list|bibliography)$/i.test(clean(h.textContent??'')));if(h){found=true;let n=h.nextElementSibling;while(n&&!/^H[1-4]$/.test(n.tagName)){sections.push(n);n=n.nextElementSibling;}}}
|
||||
for(const s of sections){const items=[...s.querySelectorAll('li')];for(const item of items.length?items:[s]){
|
||||
const title=clean(item.textContent??''),urls=[...item.querySelectorAll('a[href]')].map(a=>a.getAttribute('href')!);
|
||||
urls.push(...(title.match(/https?:\/\/[^\s<>"«»]+/gi)??[]).map(trimUrl));
|
||||
for(const raw of urls){const normalized=safeUrl(raw,url);if(normalized&&normalized!==url)refs.set(normalized,{url:normalized,title:title.slice(0,2000)});}}}
|
||||
const content=[...document.querySelectorAll('.pud__section')].map(s=>clean(s.textContent??'')).join('\n')||clean(document.querySelector('main')?.textContent??document.querySelector('h1')?.textContent??'');
|
||||
return {name:clean(document.querySelector('h1')?.textContent??''),refs:[...refs.values()],literature:found,content};}
|
||||
export async function fingerprint(v:string){return [...new Uint8Array(await crypto.subtle.digest('SHA-256',new TextEncoder().encode(v)))].map(n=>n.toString(16).padStart(2,'0')).join('');}
|
||||
export {safeUrl,publicIp,resolvePublicHost,retrieve} from './http.ts';
|
||||
export {classifyResponse as classify} from './access.ts';
|
||||
export async function checkLink(url:string):Promise<Check>{
|
||||
try{
|
||||
const signal=AbortSignal.timeout(60000);
|
||||
// Known libraries contain hidden auth/paywall templates; only rendered evidence can decide their access.
|
||||
if(provider(url))return await (await import('./browser.ts')).checkBrowser(url,signal);
|
||||
const page=await retrieve(url,false,signal),check=classifyResponse(url,page);
|
||||
if(page.code===403&&check.status==='blocked')await limitHost(new URL(page.url).hostname,403);
|
||||
if(check.evidence&&['auth','restricted'].includes(check.status))return {...check,status:'unknown',reason:'В HTML есть признаки ограничения, но доступ к полному тексту не подтверждён отрисованной страницей.',evidence:undefined};
|
||||
return check;
|
||||
}catch(e){const error=e as Error;return result(url,error instanceof HostCooldown?'blocked':error.name==='AbortError'?'timeout':'network',error instanceof HostCooldown?error.message:'Не удалось загрузить материал. Доступ к полному тексту не подтверждён.');}
|
||||
}
|
||||
+63
@@ -0,0 +1,63 @@
|
||||
import {lookup} from 'node:dns/promises';
|
||||
import {request as httpRequest,type IncomingHttpHeaders} from 'node:http';
|
||||
import {request as httpsRequest} from 'node:https';
|
||||
import {createUnzip,createBrotliDecompress} from 'node:zlib';
|
||||
import {waitForHost,limitHost} from './throttle.ts';
|
||||
|
||||
export function safeUrl(value:string,base?:string):string|null{
|
||||
try{const u=new URL(value,base),h=u.hostname.toLowerCase();
|
||||
if(!['https:','http:'].includes(u.protocol)||u.username||u.password||(u.port&&!['80','443'].includes(u.port)))return null;
|
||||
if(!h.includes('.')||h.includes(':')||/^\d+[.\d]*$/.test(h)||h.endsWith('.')||/(^|\.)(localhost|local|internal|test|invalid|example|onion|arpa)$/.test(h))return null;
|
||||
u.hash='';return u.href;
|
||||
}catch{return null;}
|
||||
}
|
||||
export function publicIp(ip:string){
|
||||
if(ip.includes(':')){const fields=ip.toLowerCase().split(':');return /^[23][0-9a-f]{3}:/i.test(ip)&&fields[0]!=='2002'&&fields[0]!=='3fff'&&!(fields[0]==='2001'&&(parseInt(fields[1]||'0',16)<=0x1ff||parseInt(fields[1],16)===0xdb8));}
|
||||
const p=ip.split('.').map(Number);if(p.length!==4||p.some(n=>!Number.isInteger(n)||n<0||n>255))return false;
|
||||
const [a,b,c]=p;return !(a===0||a===10||a===127||a>=224||a===169&&b===254||a===172&&b>=16&&b<=31||a===192&&(b===168||b===0||b===88&&c===99)||a===100&&b>=64&&b<=127||a===198&&(b===18||b===19||b===51&&c===100)||a===203&&b===0&&c===113);
|
||||
}
|
||||
export async function resolvePublicHost(host:string){
|
||||
const addresses=await lookup(host,{all:true,verbatim:true});
|
||||
if(!addresses.length)throw new Error('DNS: адрес сервера не найден');
|
||||
if(addresses.some(a=>!publicIp(a.address)))throw new Error('DNS указывает на непубличный адрес; запрос запрещён');
|
||||
return addresses.find(a=>a.family===4)??addresses[0];
|
||||
}
|
||||
export type PublicResponse={code:number;headers:IncomingHttpHeaders;body:Buffer;type:string};
|
||||
// One pinned socket, no implicit redirects/proxies; browser redirects pass this boundary again.
|
||||
export async function requestPublic(raw:string,options:{signal?:AbortSignal;maxBytes?:number;headers?:Record<string,string>;method?:'GET'|'HEAD'|'OPTIONS'}={}):Promise<PublicResponse>{
|
||||
const url=safeUrl(raw);if(!url)throw new Error('Адрес не разрешён политикой безопасности');
|
||||
const signal=AbortSignal.any([AbortSignal.timeout(12000),...(options.signal?[options.signal]:[])]);
|
||||
const aborted=new Promise<never>((_,reject)=>{if(signal.aborted)reject(new DOMException('Истекло время ожидания','AbortError'));else signal.addEventListener('abort',()=>reject(new DOMException('Истекло время ожидания','AbortError')),{once:true});});
|
||||
const target=new URL(url),ip=await Promise.race([resolvePublicHost(target.hostname),aborted]);
|
||||
return new Promise((resolve,reject)=>{
|
||||
const request=(target.protocol==='https:'?httpsRequest:httpRequest)(target,{
|
||||
agent:false,family:ip.family,signal,lookup:(_hostname,_options,callback)=>callback(null,ip.address,ip.family),method:options.method??'GET',
|
||||
headers:{'User-Agent':'Sverka/1.3 (academic bibliography access check)','Accept':'text/html,application/pdf;q=0.9,*/*;q=0.8','Accept-Encoding':'identity',...options.headers},
|
||||
},response=>{
|
||||
const code=response.statusCode??0,headers=response.headers,type=String(headers['content-type']??'');
|
||||
if(code>=300&&code<400||[429,503].includes(code)||options.method==='HEAD'){response.destroy();resolve({code,headers,type,body:Buffer.alloc(0)});return;}
|
||||
void(async()=>{
|
||||
const encoding=String(headers['content-encoding']??''),stream=/gzip|deflate/i.test(encoding)?response.pipe(createUnzip()):/br/i.test(encoding)?response.pipe(createBrotliDecompress()):response;
|
||||
response.on('error',error=>stream.destroy(error));response.on('aborted',()=>stream.destroy(new Error('Соединение прервано')));
|
||||
request.once('error',error=>{response.destroy();stream.destroy(error);});
|
||||
const chunks:Buffer[]=[];let size=0;
|
||||
try{for await(const chunk of stream){const bytes=Buffer.from(chunk);size+=bytes.length;if(size>(options.maxBytes??3000000))throw new Error('Превышен допустимый размер загрузки');chunks.push(bytes);}resolve({code,headers,type,body:Buffer.concat(chunks)});}
|
||||
catch(error){response.destroy();stream.destroy();reject(error);}
|
||||
})();
|
||||
});request.once('error',reject);request.end();
|
||||
});
|
||||
}
|
||||
export function decodeBody(body:Buffer,type:string){try{return new TextDecoder(/charset=([\w-]+)/i.exec(type)?.[1]??'utf-8').decode(body);}catch{return body.toString('utf8');}}
|
||||
export async function retrieve(raw:string,source=false,parentSignal?:AbortSignal){
|
||||
let url=safeUrl(raw);if(!url)throw new Error('Адрес не разрешён политикой безопасности');
|
||||
const signal=AbortSignal.any([AbortSignal.timeout(30000),...(parentSignal?[parentSignal]:[])]);
|
||||
for(let hop=0;hop<5;hop++){
|
||||
await waitForHost(new URL(url).hostname,signal);
|
||||
const result=await requestPublic(url,{signal,maxBytes:source?3000000:1500000});
|
||||
if(result.code!==403||source)await limitHost(new URL(url).hostname,result.code,result.headers['retry-after']);
|
||||
const location=result.headers.location;
|
||||
if(location&&result.code>=300&&result.code<400){const next=safeUrl(location,url);if(!next)throw new Error('Небезопасное перенаправление');url=next;continue;}
|
||||
return {code:result.code,url,type:result.type,body:decodeBody(result.body,result.type)};
|
||||
}
|
||||
throw new Error('Слишком много перенаправлений');
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
export const VERSION='1.3.0';
|
||||
export const PROGRAMS=[
|
||||
{id:'it',name:'Цифровые технологии и телекоммуникации',level:'Бакалавриат',url:'https://www.hse.ru/ba/it/courses'},
|
||||
{id:'is',name:'Информационная безопасность',level:'Бакалавриат',url:'https://www.hse.ru/ba/is/courses'},
|
||||
{id:'cosec',name:'Компьютерная безопасность',level:'Специалитет',url:'https://www.hse.ru/ba/cosec/courses'},
|
||||
{id:'internet',name:'Интернет вещей и киберфизические системы',level:'Магистратура',url:'https://www.hse.ru/ma/internet/courses'},
|
||||
{id:'cyber',name:'Информационная безопасность и технологии искусственного интеллекта',level:'Магистратура',url:'https://www.hse.ru/ma/cyber/courses'}];
|
||||
export type Status='ok'|'broken'|'auth'|'blocked'|'timeout'|'network'|'restricted'|'unknown';
|
||||
export const LABELS:Record<string,string>={ok:'Полный текст доступен',broken:'Недоступна: удалена',auth:'Недоступна: нужен вход',blocked:'Проверка ограничена',timeout:'Нет ответа',network:'Ошибка соединения',restricted:'Недоступна',unknown:'Доступ не подтверждён'};
|
||||
export type Check={url:string;finalUrl?:string;status:Status;reason:string;code?:number;checkedAt:string;rulesVersion?:string;profile?:'anonymous';evidence?:{text:string;url:string;rule:string}};
|
||||
export type Reference={url:string;title:string};
|
||||
export type Course={url:string;name:string;program:string;hash?:string;refs:Reference[];issue?:string;checkedAt:string;seen:string;reused:boolean;literature:boolean};
|
||||
export type Run={id:string;source?:'manual'|'schedule';status:'running'|'complete';startedAt:string;finishedAt?:string;force:boolean;phase:'catalogs'|'courses'|'links'|'done';catalogs:{url:string;program:string}[];visited:string[];courses:{url:string;name:string;program:string}[];links:string[];seen:string[];errors:{url:string;reason:string}[];processed:number;reused:number;checked:number;totalCourses:number;totalLinks:number;message:string};
|
||||
export type ScheduleSettings={enabled:boolean;frequency:'daily'|'weekly';time:string;weekday:number;force:boolean};
|
||||
export type Schedule=ScheduleSettings & {nextRunAt:string|null;updatedAt:string|null;lastRunAt:string|null;lastAttemptAt:string|null;lastOutcome:'started'|'skipped'|null;lastError:string|null};
|
||||
export const DEFAULT_SCHEDULE:Schedule={enabled:false,frequency:'daily',time:'09:00',weekday:1,force:false,nextRunAt:null,updatedAt:null,lastRunAt:null,lastAttemptAt:null,lastOutcome:null,lastError:null};
|
||||
export type Dashboard={run:Run|null;courses:Course[];checks:Record<string,Check>;programs:typeof PROGRAMS;schedule:Schedule};
|
||||
@@ -0,0 +1 @@
|
||||
export function provider(raw:string){try{const h=new URL(raw).hostname;if(h==='springernature.com'||h.endsWith('.springernature.com'))return 'springer.com';return ['lanbook.com','urait.ru','znanium.ru','ibooks.ru','springer.com','book.ru'].find(root=>h===root||h.endsWith('.'+root))??null;}catch{return null;}}
|
||||
@@ -0,0 +1,50 @@
|
||||
import {PROGRAMS,DEFAULT_SCHEDULE,type Schedule,type Run,type Course,type Check} from './model.ts';
|
||||
import {checkLink,retrieve,parseCatalog,parseCourse,fingerprint} from './checker.ts';
|
||||
import {database,transaction,type Connection} from './store.ts';
|
||||
import {checkCurrent} from './access.ts';
|
||||
async function read<T>(key:string):Promise<T|null>{const {rows}=await (await database()).query<{value:string}>('SELECT value FROM records WHERE key=$1',[key]);return rows[0]?JSON.parse(rows[0].value):null;}
|
||||
export async function getRun(connection?:Connection):Promise<Run|null>{const {rows}=await (connection??await database()).query<{data:string}>('SELECT data FROM scan WHERE id=1');return rows[0]?JSON.parse(rows[0].data):null;}
|
||||
export async function dashboard(){const run=await getRun(),{rows}=await (await database()).query<{key:string;value:string}>('SELECT key,value FROM records'),courses:Course[]=[],checks:Record<string,Check>={};
|
||||
let schedule:Schedule={...DEFAULT_SCHEDULE};for(const row of rows){if(row.key.startsWith('course:'))courses.push(JSON.parse(row.value));else if(row.key.startsWith('link:')){const check:Check=JSON.parse(row.value);checks[row.key.slice(5)]=checkCurrent(check)?check:{...check,status:'unknown',reason:'Результат устарел или получен по прежним правилам. Запустите проверку доступа к полному тексту.',evidence:undefined};}else if(row.key==='schedule')schedule=JSON.parse(row.value);}return {run,courses,checks,programs:PROGRAMS,schedule};}
|
||||
export async function start(force:boolean,source:'manual'|'schedule'='manual',connection?:Connection){const run:Run={id:crypto.randomUUID(),source,status:'running',phase:'catalogs',startedAt:new Date().toISOString(),force,catalogs:PROGRAMS.map(p=>({url:p.url,program:p.id})),visited:[],courses:[],links:[],seen:[],errors:[],processed:0,reused:0,checked:0,totalCourses:0,totalLinks:0,message:'Читаем каталоги пяти программ'};
|
||||
const db=connection??await database();await db.query("INSERT INTO scan(id,data,lease,token) VALUES(1,$1,0,'') ON CONFLICT(id) DO UPDATE SET data=excluded.data,lease=0,token='' WHERE scan.data::jsonb->>'status'!='running'",[JSON.stringify(run)]);return getRun(db);}
|
||||
export async function step(){const token=crypto.randomUUID(),now=Date.now(),db=await database();
|
||||
const {rows}=await db.query<{data:string}>('UPDATE scan SET lease=$1,token=$2 WHERE id=1 AND lease<$3 RETURNING data',[now+90000,token,now]);if(!rows[0])return getRun();const run:Run=JSON.parse(rows[0].data);const pending:{key:string;value:unknown}[]=[];let succeeded=false;
|
||||
try{if(run.status!=='running')return run;
|
||||
if(run.phase==='catalogs'){
|
||||
const task=run.catalogs.shift();if(task&&!run.visited.includes(task.url)){run.visited.push(task.url);run.message=`Читаем каталог: ${PROGRAMS.find(p=>p.id===task.program)?.name}`;
|
||||
try{const page=await retrieve(task.url,true);if(page.code!==200)throw new Error(`Каталог вернул HTTP ${page.code}`);
|
||||
const root=PROGRAMS.find(p=>p.id===task.program)!.url,parsed=parseCatalog(page.body,task.url,root);if(!parsed.courses.length)throw new Error('Курсы не найдены: каталог недоступен или его структура изменилась');
|
||||
for(const c of parsed.courses)if(!run.seen.includes(c.url)){run.seen.push(c.url);run.courses.push({...c,program:task.program});}
|
||||
for(const url of parsed.pages)if(!run.visited.includes(url)&&!run.catalogs.some(p=>p.url===url))run.catalogs.push({url,program:task.program});run.totalCourses=run.seen.length;
|
||||
}catch(e){run.errors.push({url:task.url,reason:(e as Error).message});}}
|
||||
if(!run.catalogs.length)run.phase='courses';
|
||||
}else if(run.phase==='courses'){
|
||||
const task=run.courses.shift();if(task){run.message=`Читаем курс: ${task.name}`;const old=await read<Course>('course:'+task.url);
|
||||
let course:Course={...task,refs:old?.refs??[],checkedAt:old?.checkedAt??'',seen:run.id,reused:false,literature:false};
|
||||
try{const page=await retrieve(task.url,true);if(page.code!==200)throw new Error(`Страница курса вернула HTTP ${page.code}`);
|
||||
const parsed=parseCourse(page.body,task.url);if(!parsed.name)throw new Error('Не удалось прочитать программу курса');const hash=await fingerprint(parsed.name+parsed.content+JSON.stringify(parsed.refs));
|
||||
if(!run.force&&old?.hash===hash&&!old.issue){course={...old,seen:run.id,reused:true};run.reused++;}
|
||||
else{course={...course,name:parsed.name,hash,refs:parsed.refs,literature:parsed.literature,checkedAt:new Date().toISOString()};if(!parsed.literature)course.issue='Раздел литературы не найден. Проверьте программу курса вручную.';
|
||||
}
|
||||
const keys=course.refs.map(ref=>'link:'+ref.url),{rows}=await db.query<{key:string;value:string}>('SELECT key,value FROM records WHERE key=ANY($1::text[])',[keys]);
|
||||
const checks=new Map(rows.map(row=>[row.key,JSON.parse(row.value) as Check]));
|
||||
for(const ref of course.refs)if((run.force||!checkCurrent(checks.get('link:'+ref.url)))&&!run.links.includes(ref.url))run.links.push(ref.url);
|
||||
}catch(e){course.issue=(e as Error).message;}
|
||||
pending.push({key:'course:'+task.url,value:course});run.processed++;}
|
||||
if(!run.courses.length){run.phase='links';run.totalLinks=run.links.length;}
|
||||
}else if(run.phase==='links'){
|
||||
const url=run.links.shift();run.message='Проверяем доступность материалов';if(url){const result=await checkLink(url);pending.push({key:'link:'+result.url,value:result});run.checked++;}
|
||||
if(!run.links.length){run.status='complete';run.phase='done';run.finishedAt=new Date().toISOString();run.message=run.errors.length?'Проверка завершена с ошибками чтения каталогов':'Проверка завершена';}
|
||||
}
|
||||
succeeded=true;
|
||||
}finally{
|
||||
if(succeeded){
|
||||
try{await transaction(async client=>{
|
||||
const {rows}=await client.query<{token:string}>('SELECT token FROM scan WHERE id=1 FOR UPDATE');
|
||||
if(rows[0]?.token!==token)throw new Error('Порция проверки уже выполняется другим процессом');
|
||||
for(const record of pending)await client.query('INSERT INTO records(key,value) VALUES($1,$2) ON CONFLICT(key) DO UPDATE SET value=excluded.value',[record.key,JSON.stringify(record.value)]);
|
||||
await client.query("UPDATE scan SET data=$1,lease=0,token='' WHERE id=1 AND token=$2",[JSON.stringify(run),token]);
|
||||
});}catch(error){await db.query("UPDATE scan SET lease=0,token='' WHERE id=1 AND token=$1",[token]);throw error;}
|
||||
}else await db.query("UPDATE scan SET lease=0,token='' WHERE id=1 AND token=$1",[token]);
|
||||
}return run;}
|
||||
@@ -0,0 +1,10 @@
|
||||
import {setTimeout as sleep} from 'node:timers/promises';
|
||||
import {schedulerStep,recordSchedulerError} from './scheduler.ts';
|
||||
|
||||
// The worker shares SQLite leases with the API; restarting the container resumes scheduled runs.
|
||||
for(;;){
|
||||
let busy=false;
|
||||
try{busy=await schedulerStep();}
|
||||
catch(error){console.error('Scheduled audit failed; retrying in five seconds');try{await recordSchedulerError(error);}catch(storageError){console.error('Cannot save scheduler error');}}
|
||||
await sleep(busy?350:5000);
|
||||
}
|
||||
@@ -0,0 +1,66 @@
|
||||
import {DEFAULT_SCHEDULE,type Schedule,type ScheduleSettings} from './model.ts';
|
||||
import {database,transaction,type Connection} from './store.ts';
|
||||
import {getRun,start,step} from './scanner.ts';
|
||||
|
||||
export function validateSchedule(value:unknown):ScheduleSettings{
|
||||
if(!value||typeof value!=='object')throw new RangeError('Укажите настройки расписания.');
|
||||
const s=value as Record<string,unknown>;
|
||||
if(typeof s.enabled!=='boolean'||typeof s.force!=='boolean')throw new RangeError('Укажите состояние расписания и режим кэша.');
|
||||
if(s.frequency!=='daily'&&s.frequency!=='weekly')throw new RangeError('Выберите ежедневную или еженедельную проверку.');
|
||||
if(typeof s.time!=='string'||!/^([01]\d|2[0-3]):[0-5]\d$/.test(s.time))throw new RangeError('Укажите время в формате ЧЧ:ММ.');
|
||||
if(!Number.isInteger(s.weekday)||Number(s.weekday)<0||Number(s.weekday)>6)throw new RangeError('Выберите день недели.');
|
||||
return {enabled:s.enabled,frequency:s.frequency,time:s.time,weekday:s.weekday as number,force:s.force};
|
||||
}
|
||||
const clock=new Intl.DateTimeFormat('en-CA',{timeZone:'Europe/Moscow',year:'numeric',month:'2-digit',day:'2-digit',hour:'2-digit',minute:'2-digit',second:'2-digit',hourCycle:'h23'});
|
||||
function moscowWallTime(instant:number){const parts=Object.fromEntries(clock.formatToParts(instant).map(p=>[p.type,p.value]));return Date.UTC(+parts.year,+parts.month-1,+parts.day,+parts.hour,+parts.minute,+parts.second);}
|
||||
export function nextOccurrence(settings:ScheduleSettings,now=Date.now()):string{
|
||||
const local=new Date(moscowWallTime(now)),[hour,minute]=settings.time.split(':').map(Number);
|
||||
for(let days=0;days<=7;days++){
|
||||
const wall=Date.UTC(local.getUTCFullYear(),local.getUTCMonth(),local.getUTCDate()+days,hour,minute);
|
||||
if(settings.frequency==='weekly'&&new Date(wall).getUTCDay()!==settings.weekday)continue;
|
||||
let instant=wall;for(let i=0;i<3;i++)instant+=wall-moscowWallTime(instant);
|
||||
if(instant>now)return new Date(instant).toISOString();
|
||||
}
|
||||
throw new Error('Не удалось рассчитать следующий запуск.');
|
||||
}
|
||||
export async function getSchedule(connection?:Connection):Promise<Schedule>{
|
||||
const {rows}=await (connection??await database()).query<{value:string}>("SELECT value FROM records WHERE key='schedule'");
|
||||
return rows[0]?JSON.parse(rows[0].value):{...DEFAULT_SCHEDULE};
|
||||
}
|
||||
async function persist(schedule:Schedule,client:Connection){await client.query("INSERT INTO records(key,value) VALUES('schedule',$1) ON CONFLICT(key) DO UPDATE SET value=excluded.value",[JSON.stringify(schedule)]);}
|
||||
export async function saveSchedule(value:unknown,now=Date.now()):Promise<Schedule>{
|
||||
const settings=validateSchedule(value);
|
||||
return transaction(async client=>{
|
||||
await client.query('SELECT pg_advisory_xact_lock(72847322)');
|
||||
const schedule:Schedule={...await getSchedule(client),...settings,updatedAt:new Date(now).toISOString(),nextRunAt:settings.enabled?nextOccurrence(settings,now):null};
|
||||
await persist(schedule,client);return schedule;
|
||||
});
|
||||
}
|
||||
export async function claimScheduledRun(now=Date.now()){
|
||||
await transaction(async client=>{
|
||||
await client.query('SELECT pg_advisory_xact_lock(72847322)');
|
||||
const schedule=await getSchedule(client);
|
||||
if(schedule.enabled&&schedule.nextRunAt&&Date.parse(schedule.nextRunAt)<=now){
|
||||
const run=await getRun(client);schedule.lastAttemptAt=new Date(now).toISOString();schedule.nextRunAt=nextOccurrence(schedule,now);
|
||||
if(run?.status==='running')schedule.lastOutcome='skipped';
|
||||
else{
|
||||
const started=await start(schedule.force,'schedule',client);
|
||||
if(started?.source==='schedule'){schedule.lastRunAt=new Date(now).toISOString();schedule.lastOutcome='started';schedule.lastError=null;}
|
||||
else schedule.lastOutcome='skipped';
|
||||
}
|
||||
await persist(schedule,client);
|
||||
}
|
||||
});
|
||||
}
|
||||
export async function schedulerStep(now=Date.now()){
|
||||
await claimScheduledRun(now);
|
||||
const run=await getRun();
|
||||
if(run?.source==='schedule'&&run.status==='running'){await step();return true;}
|
||||
return false;
|
||||
}
|
||||
export async function recordSchedulerError(error:unknown){
|
||||
await transaction(async client=>{
|
||||
await client.query('SELECT pg_advisory_xact_lock(72847322)');
|
||||
const schedule=await getSchedule(client);schedule.lastError='Не удалось обработать очередь. Сервер повторит попытку через пять секунд.';await persist(schedule,client);
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
import {Pool,type PoolClient} from 'pg';
|
||||
export type Connection=Pool|PoolClient;
|
||||
const runtime=globalThis as typeof globalThis & {coursesPool?:Pool;coursesReady?:Promise<void>};
|
||||
async function inTransaction<T>(pool:Pool,work:(client:PoolClient)=>Promise<T>):Promise<T>{
|
||||
const client=await pool.connect();
|
||||
try{await client.query('BEGIN');const result=await work(client);await client.query('COMMIT');return result;}
|
||||
catch(error){await client.query('ROLLBACK').catch(()=>{});throw error;}
|
||||
finally{client.release();}
|
||||
}
|
||||
export async function database():Promise<Pool>{
|
||||
if(!runtime.coursesPool){
|
||||
if(!process.env.DATABASE_URL)throw new Error('DATABASE_URL is required');
|
||||
runtime.coursesPool=new Pool({connectionString:process.env.DATABASE_URL,max:4,connectionTimeoutMillis:5000,idleTimeoutMillis:30000,statement_timeout:15000});
|
||||
runtime.coursesPool.on('error',()=>console.error('PostgreSQL connection unavailable'));
|
||||
}
|
||||
const pool=runtime.coursesPool;
|
||||
runtime.coursesReady??=inTransaction(pool,async client=>{
|
||||
await client.query('SELECT pg_advisory_xact_lock(72847321)');
|
||||
await client.query('CREATE TABLE IF NOT EXISTS records(key TEXT PRIMARY KEY NOT NULL,value TEXT NOT NULL)');
|
||||
await client.query("CREATE TABLE IF NOT EXISTS scan(id INTEGER PRIMARY KEY,data TEXT NOT NULL,lease BIGINT NOT NULL DEFAULT 0,token TEXT NOT NULL DEFAULT '')");
|
||||
});
|
||||
try{await runtime.coursesReady;}catch(error){runtime.coursesReady=undefined;throw error;}
|
||||
return pool;
|
||||
}
|
||||
export async function transaction<T>(work:(client:PoolClient)=>Promise<T>):Promise<T>{return inTransaction(await database(),work);}
|
||||
export async function closeDatabase(){const pool=runtime.coursesPool;runtime.coursesPool=undefined;runtime.coursesReady=undefined;if(pool)await pool.end();}
|
||||
@@ -0,0 +1,41 @@
|
||||
import {setTimeout as sleep} from 'node:timers/promises';
|
||||
import {transaction} from './store.ts';
|
||||
|
||||
type HostState={nextAt:number;blockedUntil:number};
|
||||
export class HostCooldown extends Error{
|
||||
constructor(until:number){super(`Запрос отложен: сервер ограничил доступ. Пауза до ${new Date(until).toISOString()}; доступность материала не подтверждена.`);}
|
||||
}
|
||||
// Persist reservations across the API and worker, including container restarts.
|
||||
async function updateHost<T>(host:string,update:(state:HostState,now:number)=>T){
|
||||
return transaction(async client=>{
|
||||
const key='throttle:'+host;
|
||||
await client.query('SELECT pg_advisory_xact_lock(hashtextextended($1,0))',[key]);
|
||||
const {rows}=await client.query<{value:string}>('SELECT value FROM records WHERE key=$1',[key]);
|
||||
const state:HostState=rows[0]?JSON.parse(rows[0].value):{nextAt:0,blockedUntil:0};
|
||||
const result=update(state,Date.now());
|
||||
await client.query('INSERT INTO records(key,value) VALUES($1,$2) ON CONFLICT(key) DO UPDATE SET value=excluded.value',[key,JSON.stringify(state)]);
|
||||
return result;
|
||||
});
|
||||
}
|
||||
export async function reserveRequest(host:string){
|
||||
return updateHost(host,(state,now)=>{
|
||||
if(state.blockedUntil>now)throw new HostCooldown(state.blockedUntil);
|
||||
const at=Math.max(now,state.nextAt);state.nextAt=at+2000;return at;
|
||||
});
|
||||
}
|
||||
export async function waitForHost(host:string,signal?:AbortSignal){
|
||||
const at=await reserveRequest(host);await sleep(Math.max(0,at-Date.now()),undefined,{signal});
|
||||
await updateHost(host,(state,now)=>{if(state.blockedUntil>now)throw new HostCooldown(state.blockedUntil);});
|
||||
}
|
||||
export function retryDelay(value:string|undefined,now:number){
|
||||
const milliseconds=value&&/^\d+$/.test(value.trim())?Number(value)*1000:value?Date.parse(value)-now:0;
|
||||
// Ignore malformed values; keep a minimum pause even if the server says zero.
|
||||
return Math.min(8640000000000000-now,Math.max(60000,Number.isNaN(milliseconds)?0:milliseconds));
|
||||
}
|
||||
export async function limitHost(host:string,code:number,retryAfter?:string){
|
||||
if(![403,429,503].includes(code))return;
|
||||
await updateHost(host,(state,now)=>{
|
||||
state.blockedUntil=Math.max(state.blockedUntil,now+(code===403?300000:retryDelay(retryAfter,now)));
|
||||
state.nextAt=Math.max(state.nextAt,state.blockedUntil);
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
import type { NextConfig } from "next";
|
||||
|
||||
const nextConfig: NextConfig = {
|
||||
output: 'standalone',
|
||||
basePath: '/courses',
|
||||
serverExternalPackages: ['linkedom','pg','playwright','playwright-core'],
|
||||
};
|
||||
|
||||
export default nextConfig;
|
||||
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"name": "sverka",
|
||||
"version": "1.3.0",
|
||||
"private": true,
|
||||
"engines": {
|
||||
"node": ">=24.0.0"
|
||||
},
|
||||
"scripts": {
|
||||
"dev": "node server/runtime.mjs --dev",
|
||||
"build": "next build --webpack",
|
||||
"start": "node server/runtime.mjs",
|
||||
"test": "node --experimental-strip-types --experimental-test-module-mocks --test tests/*.test.ts",
|
||||
"test:browser": "node --experimental-strip-types --experimental-test-module-mocks --test tests/browser.security.ts",
|
||||
"typecheck": "tsc --noEmit"
|
||||
},
|
||||
"dependencies": {
|
||||
"linkedom": "0.18.14",
|
||||
"lucide-react": "1.63.0",
|
||||
"next": "16.3.4",
|
||||
"pg": "8.16.3",
|
||||
"playwright": "1.64.1",
|
||||
"react": "19.2.6",
|
||||
"react-dom": "19.2.6"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "22.19.19",
|
||||
"@types/pg": "8.15.5",
|
||||
"@types/react": "19.2.14",
|
||||
"@types/react-dom": "19.2.3",
|
||||
"typescript": "5.9.3"
|
||||
},
|
||||
"type": "module",
|
||||
"packageManager": "pnpm@11.25.0"
|
||||
}
|
||||
Generated
+1310
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,5 @@
|
||||
allowBuilds:
|
||||
esbuild: true
|
||||
sharp: true
|
||||
unrs-resolver: true
|
||||
workerd: true
|
||||
@@ -0,0 +1 @@
|
||||
export default {plugins:{}};
|
||||
@@ -0,0 +1 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 32 32"><rect width="32" height="32" rx="8" fill="#203e32"/><g fill="none" stroke="#d9ebbb" stroke-width="2.5" stroke-linecap="round"><path d="M13 11H11a5 5 0 0 0 0 10h2M19 11h2a5 5 0 0 1 0 10h-2M12 16h8"/></g></svg>
|
||||
|
After Width: | Height: | Size: 270 B |
@@ -0,0 +1,93 @@
|
||||
Copyright 2019 The Golos Text Project Authors (https://github.com/googlefonts/golos-text)
|
||||
|
||||
This Font Software is licensed under the SIL Open Font License, Version 1.1.
|
||||
This license is copied below, and is also available with a FAQ at:
|
||||
https://scripts.sil.org/OFL
|
||||
|
||||
|
||||
-----------------------------------------------------------
|
||||
SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
|
||||
-----------------------------------------------------------
|
||||
|
||||
PREAMBLE
|
||||
The goals of the Open Font License (OFL) are to stimulate worldwide
|
||||
development of collaborative font projects, to support the font creation
|
||||
efforts of academic and linguistic communities, and to provide a free and
|
||||
open framework in which fonts may be shared and improved in partnership
|
||||
with others.
|
||||
|
||||
The OFL allows the licensed fonts to be used, studied, modified and
|
||||
redistributed freely as long as they are not sold by themselves. The
|
||||
fonts, including any derivative works, can be bundled, embedded,
|
||||
redistributed and/or sold with any software provided that any reserved
|
||||
names are not used by derivative works. The fonts and derivatives,
|
||||
however, cannot be released under any other type of license. The
|
||||
requirement for fonts to remain under this license does not apply
|
||||
to any document created using the fonts or their derivatives.
|
||||
|
||||
DEFINITIONS
|
||||
"Font Software" refers to the set of files released by the Copyright
|
||||
Holder(s) under this license and clearly marked as such. This may
|
||||
include source files, build scripts and documentation.
|
||||
|
||||
"Reserved Font Name" refers to any names specified as such after the
|
||||
copyright statement(s).
|
||||
|
||||
"Original Version" refers to the collection of Font Software components as
|
||||
distributed by the Copyright Holder(s).
|
||||
|
||||
"Modified Version" refers to any derivative made by adding to, deleting,
|
||||
or substituting -- in part or in whole -- any of the components of the
|
||||
Original Version, by changing formats or by porting the Font Software to a
|
||||
new environment.
|
||||
|
||||
"Author" refers to any designer, engineer, programmer, technical
|
||||
writer or other person who contributed to the Font Software.
|
||||
|
||||
PERMISSION & CONDITIONS
|
||||
Permission is hereby granted, free of charge, to any person obtaining
|
||||
a copy of the Font Software, to use, study, copy, merge, embed, modify,
|
||||
redistribute, and sell modified and unmodified copies of the Font
|
||||
Software, subject to the following conditions:
|
||||
|
||||
1) Neither the Font Software nor any of its individual components,
|
||||
in Original or Modified Versions, may be sold by itself.
|
||||
|
||||
2) Original or Modified Versions of the Font Software may be bundled,
|
||||
redistributed and/or sold with any software, provided that each copy
|
||||
contains the above copyright notice and this license. These can be
|
||||
included either as stand-alone text files, human-readable headers or
|
||||
in the appropriate machine-readable metadata fields within text or
|
||||
binary files as long as those fields can be easily viewed by the user.
|
||||
|
||||
3) No Modified Version of the Font Software may use the Reserved Font
|
||||
Name(s) unless explicit written permission is granted by the corresponding
|
||||
Copyright Holder. This restriction only applies to the primary font name as
|
||||
presented to the users.
|
||||
|
||||
4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
|
||||
Software shall not be used to promote, endorse or advertise any
|
||||
Modified Version, except to acknowledge the contribution(s) of the
|
||||
Copyright Holder(s) and the Author(s) or with their explicit written
|
||||
permission.
|
||||
|
||||
5) The Font Software, modified or unmodified, in part or in whole,
|
||||
must be distributed entirely under this license, and must not be
|
||||
distributed under any other license. The requirement for fonts to
|
||||
remain under this license does not apply to any document created
|
||||
using the Font Software.
|
||||
|
||||
TERMINATION
|
||||
This license becomes null and void if any of the above conditions are
|
||||
not met.
|
||||
|
||||
DISCLAIMER
|
||||
THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
||||
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
|
||||
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
|
||||
OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
|
||||
COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
|
||||
INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
|
||||
DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
|
||||
FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
|
||||
OTHER DEALINGS IN THE FONT SOFTWARE.
|
||||
@@ -0,0 +1,93 @@
|
||||
Copyright 2011 The Lora Project Authors (https://github.com/cyrealtype/Lora-Cyrillic), with Reserved Font Name "Lora".
|
||||
|
||||
This Font Software is licensed under the SIL Open Font License, Version 1.1.
|
||||
This license is copied below, and is also available with a FAQ at:
|
||||
https://scripts.sil.org/OFL
|
||||
|
||||
|
||||
-----------------------------------------------------------
|
||||
SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
|
||||
-----------------------------------------------------------
|
||||
|
||||
PREAMBLE
|
||||
The goals of the Open Font License (OFL) are to stimulate worldwide
|
||||
development of collaborative font projects, to support the font creation
|
||||
efforts of academic and linguistic communities, and to provide a free and
|
||||
open framework in which fonts may be shared and improved in partnership
|
||||
with others.
|
||||
|
||||
The OFL allows the licensed fonts to be used, studied, modified and
|
||||
redistributed freely as long as they are not sold by themselves. The
|
||||
fonts, including any derivative works, can be bundled, embedded,
|
||||
redistributed and/or sold with any software provided that any reserved
|
||||
names are not used by derivative works. The fonts and derivatives,
|
||||
however, cannot be released under any other type of license. The
|
||||
requirement for fonts to remain under this license does not apply
|
||||
to any document created using the fonts or their derivatives.
|
||||
|
||||
DEFINITIONS
|
||||
"Font Software" refers to the set of files released by the Copyright
|
||||
Holder(s) under this license and clearly marked as such. This may
|
||||
include source files, build scripts and documentation.
|
||||
|
||||
"Reserved Font Name" refers to any names specified as such after the
|
||||
copyright statement(s).
|
||||
|
||||
"Original Version" refers to the collection of Font Software components as
|
||||
distributed by the Copyright Holder(s).
|
||||
|
||||
"Modified Version" refers to any derivative made by adding to, deleting,
|
||||
or substituting -- in part or in whole -- any of the components of the
|
||||
Original Version, by changing formats or by porting the Font Software to a
|
||||
new environment.
|
||||
|
||||
"Author" refers to any designer, engineer, programmer, technical
|
||||
writer or other person who contributed to the Font Software.
|
||||
|
||||
PERMISSION & CONDITIONS
|
||||
Permission is hereby granted, free of charge, to any person obtaining
|
||||
a copy of the Font Software, to use, study, copy, merge, embed, modify,
|
||||
redistribute, and sell modified and unmodified copies of the Font
|
||||
Software, subject to the following conditions:
|
||||
|
||||
1) Neither the Font Software nor any of its individual components,
|
||||
in Original or Modified Versions, may be sold by itself.
|
||||
|
||||
2) Original or Modified Versions of the Font Software may be bundled,
|
||||
redistributed and/or sold with any software, provided that each copy
|
||||
contains the above copyright notice and this license. These can be
|
||||
included either as stand-alone text files, human-readable headers or
|
||||
in the appropriate machine-readable metadata fields within text or
|
||||
binary files as long as those fields can be easily viewed by the user.
|
||||
|
||||
3) No Modified Version of the Font Software may use the Reserved Font
|
||||
Name(s) unless explicit written permission is granted by the corresponding
|
||||
Copyright Holder. This restriction only applies to the primary font name as
|
||||
presented to the users.
|
||||
|
||||
4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
|
||||
Software shall not be used to promote, endorse or advertise any
|
||||
Modified Version, except to acknowledge the contribution(s) of the
|
||||
Copyright Holder(s) and the Author(s) or with their explicit written
|
||||
permission.
|
||||
|
||||
5) The Font Software, modified or unmodified, in part or in whole,
|
||||
must be distributed entirely under this license, and must not be
|
||||
distributed under any other license. The requirement for fonts to
|
||||
remain under this license does not apply to any document created
|
||||
using the Font Software.
|
||||
|
||||
TERMINATION
|
||||
This license becomes null and void if any of the above conditions are
|
||||
not met.
|
||||
|
||||
DISCLAIMER
|
||||
THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
||||
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
|
||||
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
|
||||
OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
|
||||
COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
|
||||
INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
|
||||
DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
|
||||
FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
|
||||
OTHER DEALINGS IN THE FONT SOFTWARE.
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,42 @@
|
||||
@font-face {
|
||||
font-family: 'Golos Text';
|
||||
font-style: normal;
|
||||
font-weight: 400;
|
||||
font-display: swap;
|
||||
src: url(/courses/fonts/font-0.woff2) format('truetype');
|
||||
}
|
||||
@font-face {
|
||||
font-family: 'Golos Text';
|
||||
font-style: normal;
|
||||
font-weight: 500;
|
||||
font-display: swap;
|
||||
src: url(/courses/fonts/font-1.woff2) format('truetype');
|
||||
}
|
||||
@font-face {
|
||||
font-family: 'Golos Text';
|
||||
font-style: normal;
|
||||
font-weight: 600;
|
||||
font-display: swap;
|
||||
src: url(/courses/fonts/font-2.woff2) format('truetype');
|
||||
}
|
||||
@font-face {
|
||||
font-family: 'Golos Text';
|
||||
font-style: normal;
|
||||
font-weight: 700;
|
||||
font-display: swap;
|
||||
src: url(/courses/fonts/font-3.woff2) format('truetype');
|
||||
}
|
||||
@font-face {
|
||||
font-family: 'Lora';
|
||||
font-style: normal;
|
||||
font-weight: 400;
|
||||
font-display: swap;
|
||||
src: url(/courses/fonts/font-4.woff2) format('truetype');
|
||||
}
|
||||
@font-face {
|
||||
font-family: 'Lora';
|
||||
font-style: normal;
|
||||
font-weight: 500;
|
||||
font-display: swap;
|
||||
src: url(/courses/fonts/font-5.woff2) format('truetype');
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
import {spawn} from 'node:child_process';
|
||||
import {fileURLToPath} from 'node:url';
|
||||
import {join,resolve} from 'node:path';
|
||||
import {cpSync} from 'node:fs';
|
||||
|
||||
const root=resolve(fileURLToPath(new URL('..',import.meta.url)));
|
||||
const dev=process.argv.includes('--dev'),standalone=process.argv.includes('--standalone');
|
||||
if(!dev&&!standalone){cpSync(join(root,'public'),join(root,'.next/standalone/public'),{recursive:true});cpSync(join(root,'.next/static'),join(root,'.next/standalone/.next/static'),{recursive:true});}
|
||||
const webArgs=standalone?['server.js']:dev?[join(root,'node_modules/next/dist/bin/next'),'dev','--webpack','-H','127.0.0.1','-p',process.env.PORT??'5173']:[join(root,'.next/standalone/server.js')];
|
||||
const children=[spawn(process.execPath,webArgs,{cwd:root,stdio:'inherit'}),spawn(process.execPath,['--experimental-strip-types',join(root,'lib/schedule-worker.ts')],{cwd:root,stdio:'inherit'})];
|
||||
let stopping=false;
|
||||
function stop(code){if(stopping)return;stopping=true;process.exitCode=code;for(const child of children)if(child.exitCode===null)child.kill('SIGTERM');const timer=setTimeout(()=>{for(const child of children)if(child.exitCode===null)child.kill('SIGKILL');},5000);timer.unref();}
|
||||
for(const child of children){child.on('error',error=>{console.error(error);stop(1);});child.on('exit',(code,signal)=>{if(!stopping)stop(code||1);});}
|
||||
process.on('SIGINT',()=>stop(0));process.on('SIGTERM',()=>stop(0));
|
||||
@@ -0,0 +1,49 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import {test} from 'node:test';
|
||||
import {accessMessage,classifyResponse,purchaseOffer,checkCurrent,result,evidenceUrl,RULES_VERSION} from '../lib/access.ts';
|
||||
const lan='https://e.lanbook.com/book/111431',urait='https://urait.ru/course-viewer/6BDE5BCD-49B3-4DCD-B6BF-F91BF5F4657E';
|
||||
const html=(body:string)=>'<html><title>Book</title><body><main>'+body+'</main></body></html>';
|
||||
const classify=(body:string,url=lan)=>classifyResponse(url,{url,body:html(body),code:200,type:'text/html'});
|
||||
const offer='<div class="book-offers__item">Доступ онлайн<div class="book-offers__price">от 156 ₽</div><span class="book-offers__ctrl" data-id="372956">В корзину</span></div>';
|
||||
|
||||
test('Lanbook denial is unavailable with an explicit login reason and evidence',()=>{
|
||||
const check=classify('<div class="lan-callout lan-callout-danger"><h4>Чтение книги недоступно</h4><p>Для доступа к чтению книги необходимо авторизоваться.</p></div>');
|
||||
assert.equal(check.status,'auth');assert.equal(check.reason,'Для чтения книги требуется авторизация.');assert.match(check.evidence!.text,/необходимо авторизоваться/);assert.equal(check.profile,'anonymous');
|
||||
});
|
||||
test('Urait requires both login and subscription; PDF prohibition alone is insufficient',()=>{
|
||||
const check=accessMessage(urait,'Чтобы получить полный доступ к курсам, авторизуйтесь на аккаунте с подпиской учебного заведения или личной подпиской.')!;
|
||||
assert.equal(check.status,'restricted');assert.equal(check.reason,'Для чтения книги требуются авторизация и подписка.');
|
||||
assert.equal(accessMessage(urait,'Оплаченный доступ к контенту предоставляется только на платформе. Скачивание контента в PDF не доступно.'),null);
|
||||
});
|
||||
test('Znanium withdrawn and priced online access are different evidence',()=>{
|
||||
assert.equal(classify('Данная публикация изъята из фонда.','https://znanium.ru/catalog/document?pid=1040849').reason,'Публикация изъята из фонда.');
|
||||
const paid=purchaseOffer('https://znanium.ru/catalog/document?id=372956',offer)!;assert.equal(paid.status,'restricted');assert.match(paid.reason,/покупка онлайн-доступа/);assert.match(paid.evidence!.url,/id=372956/);
|
||||
assert.equal(purchaseOffer('https://znanium.ru/catalog/document?id=372956',offer.replace('Доступ онлайн','Печатная книга')),null);
|
||||
assert.equal(purchaseOffer('https://znanium.ru/catalog/document?id=999',offer),null);
|
||||
assert.equal(purchaseOffer('https://znanium.ru/catalog/document?id=372956',offer.replace('156','0')),null);
|
||||
});
|
||||
test('iBooks uses the reader denial rather than a login button on the catalogue card',()=>{
|
||||
const check=accessMessage('https://ibooks.ru/bookshelf/356956/reading','Для продолжения работы с сайтом, пожалуйста, войдите в систему')!;
|
||||
assert.equal(check.status,'auth');assert.equal(classify('<button>Войти</button><h1>Linux API</h1>Аннотация','https://ibooks.ru/products/356956').status,'unknown');
|
||||
const url='https://ibooks.ru/bookshelf/356956/reading';assert.equal(classifyResponse(url,{url,body:html('Для продолжения работы с сайтом, пожалуйста, войдите в систему'),code:403,type:'text/html'}).status,'auth');
|
||||
});
|
||||
test('Springer requires an eBook purchase offer, not the paper edition price',()=>{
|
||||
const url='https://link.springer.com/book/10.1007/978-3-642-17545-9',block='<div class="app-article-access"><h2>Access this book</h2><div class="buying-option"><span>eBook</span><span class="price-amount">EUR 117.69</span><button>Buy eBook</button></div></div>';
|
||||
assert.equal(purchaseOffer(url,block)?.status,'restricted');
|
||||
assert.equal(purchaseOffer(url,block.replaceAll('eBook','Hardcover Book')),null);
|
||||
assert.equal(purchaseOffer(url,block.replace('Access this book','This is an open access book')),null);
|
||||
});
|
||||
test('hidden templates, general login, annotations and successful HTTP do not confirm access',()=>{
|
||||
for(const body of ['<h1>Book</h1>Аннотация','<header><input type="password"/>Для чтения авторизуйтесь</header><p>Book card</p>','<div hidden>Для чтения необходимо авторизоваться</div>','<div class="hidden">Для чтения необходимо авторизоваться</div>','<div style="display:none">Для чтения необходимо авторизоваться</div>','<div style="visibility: hidden !important">Для чтения необходимо авторизоваться</div>'])assert.equal(classify(body).status,'unknown');
|
||||
});
|
||||
test('old rule versions and expired checks are invalidated independently of course content',()=>{
|
||||
const now=Date.now(),fresh=result(lan,'auth','Login');assert.equal(checkCurrent(fresh,now),true);
|
||||
assert.equal(checkCurrent({...fresh,rulesVersion:undefined},now),false);assert.equal(checkCurrent({...fresh,rulesVersion:'old'},now),false);
|
||||
assert.equal(checkCurrent({...fresh,checkedAt:new Date(now-86400001).toISOString()},now),false);
|
||||
assert.equal(checkCurrent({...fresh,status:'unknown',checkedAt:new Date(now-600001).toISOString()},now),false);assert.equal(fresh.rulesVersion,RULES_VERSION);
|
||||
});
|
||||
test('evidence URLs retain book identifiers while discarding tokens and tracking',()=>assert.equal(evidenceUrl('https://znanium.ru/catalog/document?id=372956&token=secret&utm_source=x#foo'),'https://znanium.ru/catalog/document?id=372956'));
|
||||
test('BOOK.ru guest preview is unavailable even when twenty pages can be read',()=>{
|
||||
const check=accessMessage('https://book.ru/book/940586','Перейти в модуль чтения в гостевом режиме, где доступны только 20 страниц книги.')!;
|
||||
assert.equal(check.status,'restricted');assert.match(check.reason,/до 20 страниц/);assert.match(check.reason,/полный текст недоступен/);
|
||||
});
|
||||
@@ -0,0 +1,10 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import {test} from 'node:test';
|
||||
import {browserRequestAllowed} from '../lib/browser.ts';
|
||||
test('browser policy blocks other providers, internal targets, writes and heavy resources',()=>{
|
||||
for(const url of ['http://127.0.0.1/','https://127.0.0.1/','https://[::1]/','http://e.lanbook.com/','https://e.lanbook.com.evil.org/','https://urait.ru/','file:///etc/passwd','https://user:pass@e.lanbook.com/'])assert.equal(browserRequestAllowed(url,'lanbook.com','GET','document'),false,url);
|
||||
assert.equal(browserRequestAllowed('https://e.lanbook.com/book/111431','lanbook.com','POST','fetch'),false);
|
||||
assert.equal(browserRequestAllowed('https://e.lanbook.com/cover.jpg','lanbook.com','GET','image'),false);
|
||||
assert.equal(browserRequestAllowed('https://e.lanbook.com/main.js','lanbook.com','GET','script'),true);
|
||||
assert.equal(browserRequestAllowed('https://link.springernature.com/book/a','springer.com','GET','document'),true);
|
||||
});
|
||||
@@ -0,0 +1,75 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import {test,mock} from 'node:test';
|
||||
import {createServer} from 'node:http';
|
||||
import {createSocket} from 'node:dgram';
|
||||
import {chromium} from 'playwright';
|
||||
import {useTestDatabase} from './test-database.ts';
|
||||
useTestDatabase('browser');
|
||||
const launch=chromium.launch.bind(chromium);
|
||||
mock.method(chromium,'launch',async(options:Parameters<typeof chromium.launch>[0])=>{
|
||||
try{return await launch(options);}catch(error){console.error('Chromium launch diagnostic:',(error as Error).message);throw error;}
|
||||
});
|
||||
const original=await import('../lib/http.ts');
|
||||
let fixture='',proxied:string[]=[],cookieSeen=false;
|
||||
mock.module(new URL('../lib/http.ts',import.meta.url).href,{namedExports:{...original,requestPublic:async(url:string,options:{headers?:Record<string,string>})=>{
|
||||
proxied.push(url);
|
||||
if(url.endsWith('/go-book'))return {code:302,headers:{location:'/book/111431'},body:Buffer.alloc(0),type:'text/html'};
|
||||
if(url.endsWith('/session')){cookieSeen=options.headers?.cookie?.includes('anonymous_session=fixture')??false;return {code:200,headers:{'content-type':'text/plain'},body:Buffer.from('ok'),type:'text/plain'};}
|
||||
if(url.includes('/probe/'))return {code:200,headers:{'content-type':'application/javascript'},body:Buffer.from('window.loadedChunks=(window.loadedChunks||0)+1;'),type:'application/javascript'};
|
||||
if(url.endsWith('/redirect'))return {code:302,headers:{location:target},body:Buffer.alloc(0),type:'text/html'};
|
||||
if(!/^https:\/\/(e\.lanbook\.com|book\.ru)\//.test(url))throw new Error('Unexpected transport destination');
|
||||
return {code:200,headers:{'content-type':'text/html; charset=utf-8','set-cookie':['anonymous_session=fixture; Path=/; Secure; SameSite=Lax']},body:Buffer.from(fixture),type:'text/html'};
|
||||
}}});
|
||||
const {checkBrowser}=await import('../lib/browser.ts');
|
||||
let target='';
|
||||
test('sandboxed Chromium blocks private HTTP, redirects, frames, popups, workers, WebSockets and WebRTC UDP',async()=>{
|
||||
let httpHits=0,udpHits=0;
|
||||
const server=createServer((_req,res)=>{httpHits++;res.end('private');}),udp=createSocket('udp4');udp.on('message',()=>udpHits++);
|
||||
await new Promise<void>(resolve=>server.listen(0,'127.0.0.1',resolve));await new Promise<void>(resolve=>udp.bind(0,'127.0.0.1',resolve));
|
||||
const address=server.address();assert.ok(address&&typeof address!=='string');target='http://127.0.0.1:'+address.port+'/private';
|
||||
const udpAddress=udp.address();
|
||||
fixture=`<!doctype html><html><body><h1>Book</h1><div id="message">Loading</div><script>
|
||||
fetch(${JSON.stringify(target)}).catch(()=>{});
|
||||
fetch('https://e.lanbook.com/session').catch(()=>{});
|
||||
fetch('https://e.lanbook.com/redirect').catch(()=>{});
|
||||
const iframe=document.createElement('iframe');iframe.src=${JSON.stringify(target)};document.body.appendChild(iframe);
|
||||
window.open(${JSON.stringify(target)});
|
||||
new Worker(URL.createObjectURL(new Blob(['fetch('+JSON.stringify(${JSON.stringify(target)})+').catch(()=>{})'])));
|
||||
try{new WebSocket(${JSON.stringify(target.replace('http:','ws:'))})}catch{}
|
||||
try{const rtc=new RTCPeerConnection({iceServers:[{urls:'stun:127.0.0.1:${udpAddress.port}'}]});rtc.createDataChannel('test');rtc.createOffer().then(offer=>rtc.setLocalDescription(offer));}catch{}
|
||||
setTimeout(()=>document.getElementById('message').textContent='Для доступа к чтению книги необходимо авторизоваться.',2000);
|
||||
</script></body></html>`;
|
||||
try{
|
||||
const check=await checkBrowser('https://e.lanbook.com/book/111431');
|
||||
assert.equal(check.status,'auth',check.reason);assert.equal(httpHits,0);assert.equal(udpHits,0);assert.equal(cookieSeen,true,'The fresh anonymous cookie must survive the pinned transport');
|
||||
assert.ok(proxied.every(url=>url.startsWith('https://e.lanbook.com/')));assert.ok(proxied.includes('https://e.lanbook.com/redirect'));
|
||||
}finally{await new Promise<void>(resolve=>server.close(()=>resolve()));udp.close();}
|
||||
});
|
||||
test('a CSS-hidden login template does not make a known library book unavailable',async()=>{
|
||||
fixture='<html><head><style>.reader-login-template {display:none}</style></head><body><h1>Book</h1><div class="reader-login-template">Для доступа к чтению книги необходимо авторизоваться.</div><p>Only the catalogue card has loaded.</p></body></html>';
|
||||
const check=await checkBrowser('https://e.lanbook.com/book/2');assert.equal(check.status,'unknown',check.reason);assert.equal(check.code,200,'The browser must actually render the fixture');
|
||||
});
|
||||
test('a burst of essential chunks is queued instead of dropping requests over the parallelism cap',async()=>{
|
||||
fixture='<html><body><h1>Book</h1><div id="message">Loading</div><script>for(let i=0;i<16;i++){let s=document.createElement("script");s.src="/probe/"+i+".js";document.head.appendChild(s)}setInterval(()=>{if(window.loadedChunks===16)document.getElementById("message").textContent="Для доступа к чтению книги необходимо авторизоваться."},50)</script></body></html>';
|
||||
const check=await checkBrowser('https://e.lanbook.com/book/3');assert.equal(check.status,'auth',check.reason);
|
||||
});
|
||||
test('a public document redirect is explicitly revalidated and rendered at its final URL',async()=>{
|
||||
fixture='<html><body><h1>Book</h1>Для доступа к чтению книги необходимо авторизоваться.</body></html>';
|
||||
const check=await checkBrowser('https://e.lanbook.com/go-book');assert.equal(check.status,'auth',check.reason);assert.equal(check.finalUrl,'https://e.lanbook.com/book/111431');
|
||||
});
|
||||
test('BOOK.ru CSS pseudo-element tooltip proves that only a guest fragment is available',async()=>{
|
||||
fixture='<html><head><style>a::after{content:none}a:hover::after{content:"В гостевом режиме доступны только 20 страниц книги.";display:block}</style></head><body><h1>Book</h1><a href="https://reader.new.book.ru/">Ознакомиться</a></body></html>';
|
||||
const check=await checkBrowser('https://book.ru/book/940586');assert.equal(check.status,'restricted',check.reason);assert.match(check.reason,/до 20 страниц/);
|
||||
});
|
||||
|
||||
test('BOOK.ru locked orange preview is unavailable even without a loaded tooltip',async()=>{
|
||||
fixture='<html><body><h1>Book</h1><a class="book_mode_orange__fixture" href="https://reader.new.book.ru/"><span class="book_marker2__fixture">Lock</span>Ознакомиться</a></body></html>';
|
||||
const check=await checkBrowser('https://book.ru/book/940586');assert.equal(check.status,'restricted',check.reason);assert.equal(check.evidence?.rule,'bookru-locked-preview');assert.doesNotMatch(check.reason,/20/);
|
||||
fixture=fixture.replace('</body>','<a href="https://reader.new.book.ru/">Читать полностью</a></body>');
|
||||
assert.equal((await checkBrowser('https://book.ru/book/940586')).status,'unknown','A full-reading control contradicts the locked-preview-only signal');
|
||||
});
|
||||
|
||||
test('a hidden BOOK.ru preview cannot prove a restriction through its pseudo-element',async()=>{
|
||||
fixture='<html><head><style>a::after{content:"В гостевом режиме доступны только 20 страниц книги.";display:block}</style></head><body><h1>Book</h1><div style="display:none"><a href="https://reader.new.book.ru/">Ознакомиться</a></div></body></html>';
|
||||
const check=await checkBrowser('https://book.ru/book/940586');assert.equal(check.status,'unknown',check.reason);assert.equal(check.code,200);
|
||||
});
|
||||
@@ -0,0 +1,19 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import {readFileSync} from 'node:fs';
|
||||
import {test} from 'node:test';
|
||||
import {parseCatalog,parseCourse,classify,safeUrl,fingerprint} from '../lib/checker.ts';
|
||||
const url='https://www.hse.ru/ba/is/courses/1.html';
|
||||
const result=(body='',code=200,finalUrl='https://public.org/book')=>({body,code,url:finalUrl,type:'text/html'});
|
||||
const html='<html><head><title>Book</title></head><body><h1>Algorithms</h1><div class="pud__section"><h4>Список литературы</h4><ul><li>Книга https://books.org/a. <a href="https://books.org/a">читать</a></li><li>Другая https://books.org/b (дата).</li></ul></div><footer><a href="https://ignore.org/">footer</a></footer></body></html>';
|
||||
test('extracts linked and plain-text bibliography, normalizes and excludes footer',()=>{const p=parseCourse(html,url);assert.equal(p.name,'Algorithms');assert.deepEqual(p.refs.map(r=>r.url),['https://books.org/a','https://books.org/b']);assert.equal(p.literature,true);});
|
||||
test('missing bibliography is explicit',()=>assert.equal(parseCourse('<h1>Course</h1>',url).literature,false));
|
||||
test('pagination follows current program and excludes archived years',()=>{const root='https://www.hse.ru/ba/is/courses',p=parseCatalog('<a href="/ba/is/courses/1.html">A</a><a href="/ba/is/courses/page2.html?year=2026">2</a><a href="/ba/is/courses?year=2023">archive</a><a href="/ba/it/courses/4.html">other</a>',root,root);assert.equal(p.courses.length,1);assert.deepEqual(p.pages,[root+'/page2.html?year=2026']);});
|
||||
test('classifies errors, login, protection and subscription separately',()=>{assert.equal(classify(url,result('',404)).status,'broken');assert.equal(classify(url,result('',403)).status,'blocked');assert.equal(classify(url,result('',401)).status,'auth');assert.equal(classify(url,result('',200,'https://public.org/login')).status,'auth');assert.equal(classify(url,result('<html><title>Just a moment</title><body>Checking your browser</body></html>')).status,'blocked');assert.equal(classify(url,result('<html><body>Sign in to read this book</body></html>')).status,'auth');assert.equal(classify(url,result('<html><body><input type="password"/></body></html>')).status,'unknown');assert.equal(classify(url,result('<html><body>Open book</body></html>')).status,'unknown');});
|
||||
test('rejects dangerous and non-http URLs',()=>{for(const v of ['http://127.0.0.1','http://[::1]','http://localhost/a','https://foo.internal','file:///a','https://a:b@public.org','https://public.org:8080'])assert.equal(safeUrl(v),null);assert.equal(safeUrl('https://public.org/book#x'),'https://public.org/book');});
|
||||
test('hash detects content changes and remains stable',async()=>{assert.equal(await fingerprint(html),await fingerprint(html));assert.notEqual(await fingerprint(html),await fingerprint(html+'x'));});
|
||||
import {publicIp,trimUrl} from '../lib/checker.ts';
|
||||
test('preserves balanced parentheses in URLs',()=>{assert.equal(trimUrl('https://public.org/Foo_(bar).'),'https://public.org/Foo_(bar)');assert.equal(trimUrl('https://public.org/book).'),'https://public.org/book');});
|
||||
test('rejects private, reserved and loopback DNS answers',()=>{for(const ip of ['127.0.0.1','10.1.2.3','169.254.169.254','100.64.0.1','172.16.2.2','192.168.1.1','192.0.2.1','198.18.1.1','203.0.113.1','::1','fc00::1','fe80::1','2001:db8::1'])assert.equal(publicIp(ip),false,ip);assert.equal(publicIp('1.1.1.1'),true);assert.equal(publicIp('2606:4700:4700::1111'),true);});
|
||||
test('does not mark a JavaScript loading shell as accessible',()=>{assert.equal(classify(url,result('<html><body><div>Загрузка...</div><script src="app.js"></script></body></html>')).status,'unknown');});
|
||||
test('empty HTML and No Content responses do not prove material access',()=>{for(const code of [200,204,205])assert.equal(classify(url,result('',code)).status,'unknown');});
|
||||
test('non-successful binary responses do not count as available material',()=>{for(const code of [302,304])assert.equal(classify(url,{...result('',code),type:'application/pdf'}).status,'unknown');assert.equal(classify(url,{...result(),type:'application/pdf'}).status,'unknown');assert.equal(classify(url,{...result('%PDF-1.4 book file\n%%EOF'),type:'application/pdf'}).status,'unknown');});
|
||||
@@ -0,0 +1,21 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import {test,mock} from 'node:test';
|
||||
import {EventEmitter} from 'node:events';
|
||||
import {Readable} from 'node:stream';
|
||||
let connections=0,pinned='',code=200;
|
||||
mock.module('node:dns/promises',{namedExports:{lookup:async(host:string)=>[{address:host.startsWith('private')?'127.0.0.1':'1.1.1.1',family:4}]}});
|
||||
mock.module('node:https',{namedExports:{request:(_url:URL,options:{lookup:(h:string,o:object,cb:(e:unknown,ip:string)=>void)=>void},callback:(r:unknown)=>void)=>{
|
||||
connections++;options.lookup('unused',{},(_e,ip)=>{pinned=ip;});
|
||||
const request=new EventEmitter() as EventEmitter&{end:()=>void};request.end=()=>queueMicrotask(()=>{const response=Readable.from(['<html>Book</html>']) as Readable&{statusCode:number;headers:object};response.statusCode=code;response.headers={'content-type':'text/html',location:code===302?'http://127.0.0.1/private':undefined};callback(response);});return request;
|
||||
}}});
|
||||
const {requestPublic}=await import('../lib/http.ts');
|
||||
test('outbound transport rejects unsafe URLs and private DNS before creating a socket',async()=>{
|
||||
for(const url of ['http://127.0.0.1/','http://[::1]/','file:///etc/passwd','https://private.provider.org/'])await assert.rejects(requestPublic(url));
|
||||
assert.equal(connections,0);
|
||||
});
|
||||
test('a public DNS address is pinned into the socket lookup',async()=>{
|
||||
const response=await requestPublic('https://public.provider.org/book');assert.equal(pinned,'1.1.1.1');assert.equal(response.code,200);assert.equal(connections,1);
|
||||
});
|
||||
test('a redirect is returned to the caller without following it past the trust boundary',async()=>{
|
||||
code=302;const before=connections,response=await requestPublic('https://public.provider.org/redirect');assert.equal(response.code,302);assert.equal(response.headers.location,'http://127.0.0.1/private');assert.equal(connections,before+1);
|
||||
});
|
||||
@@ -0,0 +1,15 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import {test} from 'node:test';
|
||||
import {POST} from '../app/api/audit/route.ts';
|
||||
const request=(origin:string,host:string)=>new Request('http://localhost:5173/courses/api/audit',{method:'POST',headers:{Origin:origin,Host:host,'Content-Type':'application/json'},body:JSON.stringify({action:'invalid'})});
|
||||
test('development accepts the actual browser Host, not Next internal localhost',async()=>{
|
||||
delete process.env.COURSES_ORIGIN;
|
||||
assert.equal((await POST(request('http://127.0.0.1:5173','127.0.0.1:5173'))).status,400);
|
||||
assert.equal((await POST(request('https://other.example','127.0.0.1:5173'))).status,403);
|
||||
});
|
||||
test('canonical production origin works behind a proxy and rejects another site',async()=>{
|
||||
process.env.COURSES_ORIGIN='https://miem.space';
|
||||
assert.equal((await POST(request('https://miem.space','internal:3000'))).status,400);
|
||||
assert.equal((await POST(request('https://other.example','internal:3000'))).status,403);
|
||||
delete process.env.COURSES_ORIGIN;
|
||||
});
|
||||
@@ -0,0 +1,40 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import {test} from 'node:test';
|
||||
import {database} from '../lib/store.ts';
|
||||
import {dashboard,getRun,start,step} from '../lib/scanner.ts';
|
||||
import type {Run} from '../lib/model.ts';
|
||||
import {result} from '../lib/access.ts';
|
||||
import {useTestDatabase} from './test-database.ts';
|
||||
useTestDatabase('scanner');
|
||||
test('only one shared manual run can be started',async()=>{
|
||||
const [first,second]=await Promise.all([start(false),start(false)]);
|
||||
assert.equal(first?.id,second?.id);assert.equal(second?.force,false);
|
||||
});
|
||||
test('a failed record transaction preserves its queue and can resume',async()=>{
|
||||
const run=(await getRun())!;run.phase='links';run.links=['http://127.0.0.1/'];run.totalLinks=1;run.checked=0;
|
||||
const db=await database();await db.query("UPDATE scan SET data=$1,lease=0,token='' WHERE id=1",[JSON.stringify(run)]);
|
||||
await db.query("CREATE FUNCTION reject_record() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN RAISE EXCEPTION 'simulated disk failure'; END $$");
|
||||
await db.query('CREATE TRIGGER fail_record BEFORE INSERT ON records FOR EACH ROW EXECUTE FUNCTION reject_record()');
|
||||
try{await assert.rejects(step(),/simulated disk failure/);const afterFailure=(await getRun())!;assert.deepEqual(afterFailure.links,run.links);assert.equal(afterFailure.checked,0);assert.equal((await dashboard()).checks['http://127.0.0.1/'],undefined);}
|
||||
finally{await db.query('DROP TRIGGER fail_record ON records');await db.query('DROP FUNCTION reject_record()');}
|
||||
const resumed=(await step())!;assert.equal(resumed.status,'complete');assert.equal(resumed.checked,1);assert.equal((await dashboard()).checks['http://127.0.0.1/'].status,'network');
|
||||
});
|
||||
test('an active lease prevents two workers from processing the same queue',async()=>{
|
||||
const run=(await start(false))!;run.phase='links';run.links=['http://127.0.0.1/'];
|
||||
await (await database()).query('UPDATE scan SET data=$1,lease=$2,token=$3 WHERE id=1',[JSON.stringify(run),Date.now()+60000,'another-worker']);
|
||||
const unchanged=(await step()) as Run;assert.deepEqual(unchanged.links,run.links);assert.equal(unchanged.checked,0);
|
||||
});
|
||||
|
||||
test('a step consumes only one bibliography link',async()=>{
|
||||
const db=await database(),run=(await getRun())!;run.phase='links';run.links=['http://127.0.0.1/a','http://127.0.0.1/b'];run.checked=0;
|
||||
await db.query("UPDATE scan SET data=$1,lease=0,token='' WHERE id=1",[JSON.stringify(run)]);
|
||||
const after=(await step())!;assert.equal(after.checked,1);assert.deepEqual(after.links,['http://127.0.0.1/b']);assert.equal(after.status,'running');
|
||||
});
|
||||
|
||||
test('previous HTTP-only results are not presented as available after a rules upgrade',async()=>{
|
||||
const db=await database(),old={url:'https://books.org/old',status:'ok',reason:'HTTP 200',checkedAt:new Date().toISOString()};
|
||||
await db.query('INSERT INTO records(key,value) VALUES($1,$2)', ['link:'+old.url,JSON.stringify(old)]);
|
||||
assert.equal((await dashboard()).checks[old.url].status,'unknown');
|
||||
const current=result(old.url,'auth','Для чтения книги требуется авторизация.');await db.query('UPDATE records SET value=$1 WHERE key=$2',[JSON.stringify(current),'link:'+old.url]);
|
||||
assert.equal((await dashboard()).checks[old.url].status,'auth');
|
||||
});
|
||||
@@ -0,0 +1,72 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import {test} from 'node:test';
|
||||
import {fileURLToPath} from 'node:url';
|
||||
import {spawn} from 'node:child_process';
|
||||
import {setTimeout as sleep} from 'node:timers/promises';
|
||||
import {database} from '../lib/store.ts';
|
||||
import {DEFAULT_SCHEDULE} from '../lib/model.ts';
|
||||
import {getRun,start,dashboard} from '../lib/scanner.ts';
|
||||
import {nextOccurrence,validateSchedule,saveSchedule,getSchedule,claimScheduledRun} from '../lib/scheduler.ts';
|
||||
import {POST} from '../app/api/audit/route.ts';
|
||||
|
||||
import {useTestDatabase} from './test-database.ts';
|
||||
useTestDatabase('scheduler');
|
||||
const settings={...DEFAULT_SCHEDULE,enabled:true};
|
||||
const now=Date.parse('2026-10-29T05:00:00Z');
|
||||
async function reset(){const db=await database();await db.query('DELETE FROM records');await db.query('DELETE FROM scan');}
|
||||
|
||||
test('daily and weekly times use Moscow including midnight, month rollover and exact boundary',()=>{
|
||||
assert.equal(nextOccurrence(settings,now),'2026-10-29T06:00:00.000Z');
|
||||
assert.equal(nextOccurrence(settings,Date.parse('2026-10-29T06:00:00Z')),'2026-10-30T06:00:00.000Z');
|
||||
assert.equal(nextOccurrence({...settings,time:'00:15'},Date.parse('2026-10-31T21:00:00Z')),'2026-10-31T21:15:00.000Z');
|
||||
assert.equal(nextOccurrence({...settings,frequency:'weekly',weekday:1},now),'2026-11-02T06:00:00.000Z');
|
||||
});
|
||||
test('strict validation rejects impossible times and malformed configuration',()=>{
|
||||
for(const bad of [null,{}, {...settings,time:'25:00'},{...settings,time:'9:00'},{...settings,time:'12:90'},{...settings,enabled:'true'},{...settings,force:1},{...settings,weekday:7},{...settings,weekday:1.5},{...settings,frequency:'hourly'}])assert.throws(()=>validateSchedule(bad),RangeError);
|
||||
});
|
||||
test('schedule can be created, edited and disabled without dropping history',async()=>{
|
||||
await reset();assert.equal((await getSchedule()).enabled,false);
|
||||
const first=await saveSchedule(settings,now);assert.equal(first.nextRunAt,'2026-10-29T06:00:00.000Z');
|
||||
const changed=await saveSchedule({...settings,time:'11:30',frequency:'weekly',weekday:5,force:true},now);
|
||||
assert.equal(changed.nextRunAt,'2026-10-30T08:30:00.000Z');assert.equal((await dashboard()).schedule.force,true);
|
||||
const disabled=await saveSchedule({...changed,enabled:false},now);assert.equal(disabled.nextRunAt,null);assert.equal((await getSchedule()).time,'11:30');
|
||||
});
|
||||
test('a due slot is claimed once and persists a scheduled source and cache choice',async()=>{
|
||||
await reset();await saveSchedule({...settings,force:true},now);
|
||||
const due=Date.parse('2026-10-29T06:00:00Z');await Promise.all([claimScheduledRun(due),claimScheduledRun(due)]);
|
||||
const run=await getRun();assert.equal(run?.source,'schedule');assert.equal(run?.force,true);
|
||||
const id=run?.id;await claimScheduledRun(due);assert.equal((await getRun())?.id,id);
|
||||
assert.equal((await getSchedule()).lastOutcome,'started');assert.equal((await getSchedule()).nextRunAt,'2026-10-30T06:00:00.000Z');
|
||||
});
|
||||
test('an existing manual run is preserved and the conflicting slot is skipped',async()=>{
|
||||
await reset();await saveSchedule(settings,now);const manual=await start(false);
|
||||
await claimScheduledRun(Date.parse('2026-10-29T06:00:00Z'));
|
||||
assert.equal((await getRun())?.id,manual?.id);assert.equal((await getRun())?.source,'manual');assert.equal((await getSchedule()).lastOutcome,'skipped');
|
||||
});
|
||||
test('downtime catches up once rather than replaying every missed day',async()=>{
|
||||
await reset();await saveSchedule(settings,now);const later=Date.parse('2026-11-05T08:00:00Z');
|
||||
await claimScheduledRun(later);const id=(await getRun())?.id;await claimScheduledRun(later);
|
||||
assert.equal((await getRun())?.id,id);assert.equal((await getSchedule()).nextRunAt,'2026-11-06T06:00:00.000Z');
|
||||
});
|
||||
test('failed schedule claim rolls back both the new run and the consumed slot',async()=>{
|
||||
await reset();await saveSchedule(settings,now);const original=(await getSchedule());
|
||||
const db=await database();await db.query("CREATE FUNCTION reject_schedule() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN RAISE EXCEPTION 'schedule disk failure'; END $$");
|
||||
await db.query("CREATE TRIGGER fail_schedule BEFORE UPDATE ON records FOR EACH ROW WHEN (NEW.key='schedule') EXECUTE FUNCTION reject_schedule()");
|
||||
try{await assert.rejects(claimScheduledRun(Date.parse('2026-10-29T06:00:00Z')),/schedule disk failure/);assert.equal(await getRun(),null);assert.deepEqual((await getSchedule()),original);}
|
||||
finally{await db.query('DROP TRIGGER fail_schedule ON records');await db.query('DROP FUNCTION reject_schedule()');}
|
||||
});
|
||||
test('schedule API rejects a foreign Origin and invalid input, then persists valid settings',async()=>{
|
||||
await reset();process.env.COURSES_ORIGIN='https://miem.space';
|
||||
const request=(origin:string,schedule:unknown)=>new Request('http://internal:3000/courses/api/audit',{method:'POST',headers:{Origin:origin,'Content-Type':'application/json'},body:JSON.stringify({action:'schedule',schedule})});
|
||||
try{assert.equal((await POST(request('https://other.example',settings))).status,403);assert.equal((await POST(request('https://miem.space',{...settings,time:'99:99'}))).status,400);assert.equal((await POST(request('https://miem.space',settings))).status,200);assert.equal((await getSchedule()).enabled,true);}
|
||||
finally{delete process.env.COURSES_ORIGIN;}
|
||||
});
|
||||
test('the standalone worker resumes a saved scheduled run without any browser or API client',async()=>{
|
||||
await reset();const run=(await start(false,'schedule'))!;run.phase='links';run.links=['http://127.0.0.1/'];run.totalLinks=1;
|
||||
await (await database()).query('UPDATE scan SET data=$1 WHERE id=1',[JSON.stringify(run)]);
|
||||
const worker=spawn(process.execPath,['--experimental-strip-types',fileURLToPath(new URL('../lib/schedule-worker.ts',import.meta.url))],{env:{...process.env},stdio:'ignore'});
|
||||
const exited=new Promise<void>(resolve=>worker.once('exit',()=>resolve()));
|
||||
try{const deadline=Date.now()+8000;while((await getRun())?.status!=='complete'&&Date.now()<deadline)await sleep(50);
|
||||
assert.equal((await getRun())?.status,'complete');assert.equal((await getRun())?.id,run.id);assert.equal((await dashboard()).checks['http://127.0.0.1/'].status,'network');
|
||||
}finally{worker.kill('SIGTERM');await exited;}
|
||||
});
|
||||
@@ -0,0 +1,23 @@
|
||||
import {before,after} from 'node:test';
|
||||
import {randomUUID} from 'node:crypto';
|
||||
import {Pool} from 'pg';
|
||||
import {closeDatabase} from '../lib/store.ts';
|
||||
|
||||
export function useTestDatabase(label:string){
|
||||
let url:string,schema:string;
|
||||
before(async()=>{
|
||||
if(!process.env.TEST_DATABASE_URL)throw new Error('Tests require a dedicated local TEST_DATABASE_URL');
|
||||
const target=new URL(process.env.TEST_DATABASE_URL);
|
||||
if(!['127.0.0.1','localhost'].includes(target.hostname)||target.pathname!=='/sverka_test')throw new Error('Tests must use the isolated local sverka_test database');
|
||||
schema='sverka_test_'+label+'_'+randomUUID().replaceAll('-','');url=target.href;
|
||||
const admin=new Pool({connectionString:url});
|
||||
try{await admin.query(`CREATE SCHEMA "${schema}"`);}finally{await admin.end();}
|
||||
target.searchParams.set('options','-c search_path='+schema);process.env.DATABASE_URL=target.href;
|
||||
});
|
||||
after(async()=>{
|
||||
await closeDatabase();
|
||||
if(!url||!schema)return;
|
||||
const admin=new Pool({connectionString:url});
|
||||
try{await admin.query(`DROP SCHEMA "${schema}" CASCADE`);}finally{await admin.end();}
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import {test} from 'node:test';
|
||||
import {database,closeDatabase} from '../lib/store.ts';
|
||||
import {reserveRequest,limitHost,retryDelay,waitForHost,HostCooldown} from '../lib/throttle.ts';
|
||||
import {checkLink} from '../lib/checker.ts';
|
||||
import {useTestDatabase} from './test-database.ts';
|
||||
useTestDatabase('throttle');
|
||||
|
||||
test('concurrent reservations space one host by two seconds; other hosts are independent',async()=>{
|
||||
const slots=(await Promise.all([reserveRequest('one.org'),reserveRequest('one.org'),reserveRequest('one.org')])).sort((a,b)=>a-b);
|
||||
for(let i=1;i<slots.length;i++)assert.equal(slots[i]-slots[i-1],2000);
|
||||
assert.ok(await reserveRequest('two.org')<slots[1]);
|
||||
});
|
||||
test('Retry-After handles seconds, dates, invalid and excessive values',()=>{
|
||||
const now=Date.parse('2026-10-29T00:00:00Z');
|
||||
assert.equal(retryDelay('120',now),120000);assert.equal(retryDelay('Thu, 29 Oct 2026 00:02:00 GMT',now),120000);
|
||||
for(const value of [undefined,'garbage','0','-1','Wed, 28 Oct 2026 00:00:00 GMT'])assert.equal(retryDelay(value,now),60000);
|
||||
assert.equal(retryDelay('172800',now),172800000);
|
||||
for(const value of ['99999999999999999999999999','9'.repeat(400)])assert.equal(new Date(now+retryDelay(value,now)).toISOString(),'+275760-09-13T00:00:00.000Z');
|
||||
});
|
||||
test('cooldown survives reconnect, cannot be shortened and prevents a network attempt',async()=>{
|
||||
await limitHost('limited.org',429,'600');await limitHost('limited.org',503,'1');await closeDatabase();
|
||||
await assert.rejects(waitForHost('limited.org'),HostCooldown);
|
||||
const result=await checkLink('https://limited.org/book');assert.equal(result.status,'blocked');assert.match(result.reason,/Пауза до/);
|
||||
const {rows}=await (await database()).query("SELECT value FROM records WHERE key='throttle:limited.org'");
|
||||
assert.ok(JSON.parse(rows[0].value).blockedUntil>Date.now()+590000);
|
||||
});
|
||||
test('403 backs off for five minutes and successful responses do not block hosts',async()=>{
|
||||
await limitHost('forbidden.org',403);await assert.rejects(reserveRequest('forbidden.org'),HostCooldown);
|
||||
await limitHost('available.org',200);await reserveRequest('available.org');
|
||||
});
|
||||
test('expired cooldown permits requests and waiting is abortable',async()=>{
|
||||
await limitHost('expired.org',429);await (await database()).query("UPDATE records SET value=$1 WHERE key='throttle:expired.org'",[JSON.stringify({nextAt:0,blockedUntil:Date.now()-1})]);
|
||||
await reserveRequest('expired.org');
|
||||
const controller=new AbortController();controller.abort();await assert.rejects(waitForHost('expired.org',controller.signal),{name:'AbortError'});
|
||||
});
|
||||
test('the shared gate actually waits before allowing a second request',async()=>{
|
||||
await waitForHost('paced.org');const started=Date.now();await waitForHost('paced.org');assert.ok(Date.now()-started>=1900);
|
||||
});
|
||||
@@ -0,0 +1,59 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2017",
|
||||
"lib": [
|
||||
"dom",
|
||||
"dom.iterable",
|
||||
"esnext"
|
||||
],
|
||||
"allowJs": true,
|
||||
"skipLibCheck": true,
|
||||
"strict": true,
|
||||
"noEmit": true,
|
||||
"esModuleInterop": true,
|
||||
"module": "esnext",
|
||||
"moduleResolution": "bundler",
|
||||
"resolveJsonModule": true,
|
||||
"isolatedModules": true,
|
||||
"jsx": "react-jsx",
|
||||
"incremental": true,
|
||||
"plugins": [
|
||||
{
|
||||
"name": "next"
|
||||
}
|
||||
],
|
||||
"paths": {
|
||||
"@/*": [
|
||||
"./*"
|
||||
]
|
||||
},
|
||||
"types": [
|
||||
"node"
|
||||
],
|
||||
"allowImportingTsExtensions": true
|
||||
},
|
||||
"include": [
|
||||
"next-env.d.ts",
|
||||
"app/**/*.ts",
|
||||
"app/**/*.tsx",
|
||||
"lib/model.ts",
|
||||
"lib/checker.ts",
|
||||
"lib/scanner.ts",
|
||||
"lib/store.ts",
|
||||
"tests/**/*.ts",
|
||||
".next/types/**/*.ts",
|
||||
".next/dev/types/**/*.ts",
|
||||
"lib/scheduler.ts",
|
||||
"lib/schedule-worker.ts"
|
||||
],
|
||||
"exclude": [
|
||||
"node_modules",
|
||||
"examples",
|
||||
"dist",
|
||||
".vinext",
|
||||
".wrangler",
|
||||
".sites-runtime",
|
||||
"outputs",
|
||||
"work"
|
||||
]
|
||||
}
|
||||
Reference in new issue
Block a user