From 969e229cedaf94cc745fb4594070c8843c210f55 Mon Sep 17 00:00:00 2001 From: colin Date: Wed, 20 Aug 2025 18:55:33 -0400 Subject: [PATCH] docs: add DEMO.md and DEMO-SLIDES.md; align slides with next steps (vector search, DRP/BCP); simplify diagrams for compatibility --- DEMO-SLIDES.md | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++ DEMO.md | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 132 insertions(+) create mode 100644 DEMO-SLIDES.md create mode 100644 DEMO.md diff --git a/DEMO-SLIDES.md b/DEMO-SLIDES.md new file mode 100644 index 0000000..f935e52 --- /dev/null +++ b/DEMO-SLIDES.md @@ -0,0 +1,66 @@ +## Project Ploughshares — OSINT Intake Workflow +OSINT intake for international trade signals to help detect embargo‑evasion. + +--- + +### Problem +- Fragmented intake +- Missed signals at scale +- Limited auditability + +--- + +### Approach +- Unified intake + approval (Flask UI + JSON API) +- Automated ingestion (crawlers, curated feeds) +- Quality & accessibility checks in dev + +--- + +### Architecture +```mermaid +flowchart LR + Crawler[Crawlers & Feeds] -->|POST /ingest| API + Analyst -->|Submit| WebUI + WebUI -->|REST| API + API --> DB[(DB)] + API --> Queue[Approval Queue] + Queue -->|Approve/Deny| API +``` + +--- + +### End-to-End +```mermaid +sequenceDiagram + participant Analyst + participant WebUI as Web UI + participant API + participant DB as DB + participant Crawler + + Analyst->>WebUI: Submit signal + WebUI->>API: Create (pending) + API->>DB: Persist + Crawler->>API: Ingest leads + API->>DB: Upsert + Analyst->>WebUI: Approve/Deny + WebUI->>API: Decision + API->>DB: Finalize +``` + +--- + +### Implemented +- Flask app (`docker/ploughshares/app.py`) +- Schema/init (`docker/ploughshares/schema.sql`, `init_db.py`) +- Crawlers/feeds (`docker/crawler/*`, `docker/crawler_dorks/*`) +- Quality checks (`check_accessibility.py`, `check_contrast.py`, `run_lighthouse_test.sh`) +- Tests (`tests/`) + +--- + +### Next +- Implement vector search for intake to perform semantic matching beyond literal keywords and synonyms, surfacing adjacent signals and concepts. +- Author and test Disaster Recovery Plan (DRP) and Business Continuity Plan (BCP), with RTO/RPO targets, backups, failover, and runbooks. +- Expand API docs and sample payloads. \ No newline at end of file diff --git a/DEMO.md b/DEMO.md new file mode 100644 index 0000000..411856a --- /dev/null +++ b/DEMO.md @@ -0,0 +1,66 @@ +## Project Ploughshares - OSINT Intake Workflow Demo + +This demo summarizes the initial implementation of a replacement intake workflow for OSINT signals related to international trade, intended to help detect and deter embargo-evasion patterns. The system focuses on capturing OSINT leads, triaging and approving them, and maintaining an auditable record suitable for collaboration with researchers and analysts. + +### Goals +- **Streamline OSINT intake**: Provide a clear path to submit, normalize, review, and approve OSINT signals and related evidence. +- **Create an auditable workflow**: Every step - from ingestion to analyst approval - is persisted with status and metadata. +- **Enable automation**: Support ingestion of leads from crawlers and curated feeds to reduce manual toil. +- **Improve quality**: Bake in accessibility and quality checks (contrast, Lighthouse, basic headers) for the analyst-facing UI. + +### What we built (repo tour) +- **Web/API app**: Flask application for UI and JSON endpoints + - `docker/ploughshares/app.py`, templates in `docker/ploughshares/templates/`, static assets in `docker/ploughshares/static/` + - Database schema and setup in `docker/ploughshares/schema.sql` and `docker/ploughshares/init_db.py` +- **Ingestion**: Crawler components to fetch and submit OSINT leads to the API + - `docker/crawler/*.py` and `docker/crawler_dorks/*.py` (e.g., `write_to_api.py`, `marketline_crawler.py`) +- **Quality/Accessibility**: Utility scripts and checks + - `check_accessibility.py`, `check_contrast.py`, `run_lighthouse_test.sh` +- **Tests and code quality**: `tests/` (API behavior, dependencies, and basic quality gates) +- **Runtime**: Docker Compose files for local development and deployment stacks + - `docker-compose.dev.yml`, `docker-compose.yml`, `stack.staging.yml`, `stack.production.yml` + +### High-level workflow (sequence) +```mermaid +sequenceDiagram + participant Analyst + participant WebUI as Web UI (Flask) + participant API as API (Flask/JSON) + participant DB as Database + participant Crawler + + Analyst->>WebUI: Submit OSINT signal for review + WebUI->>API: POST /transactions (signal + metadata) + API->>DB: Insert pending transaction with audit fields + + Crawler->>API: POST /ingest (scraped leads) + API->>DB: Upsert leads and link evidence + + Analyst->>WebUI: Review pending approvals + WebUI->>API: PATCH /transactions/:id approve/deny + API->>DB: Update status and record decision +``` + +### Quality gates and accessibility checks (sequence) +```mermaid +sequenceDiagram + participant Dev + participant CI as CI/Local Checks + participant LHT as Lighthouse + participant A11y as Accessibility + + Dev->>CI: Push or run checks locally + CI->>LHT: run_lighthouse_test.sh (performance & a11y) + CI->>A11y: check_accessibility.py / check_contrast.py + CI->>Dev: Reports and action items +``` + +### How this supports Project Ploughshares +- **Mission-aligned intake**: The workflow is designed to help researchers and analysts surface potential embargo-evasion signals in international trade. +- **Traceability**: Signals and decisions are stored with clear statuses for collaboration and later analysis. +- **Extensible feeds**: Crawlers and curated feeds can be extended as new sources become relevant. + +### Next steps (priority first) +- Implement vector search for intake to perform semantic matching beyond literal keywords and synonyms, surfacing adjacent signals and concepts. +- Author and test Disaster Recovery Plan (DRP) and Business Continuity Plan (BCP), with RTO/RPO targets, backups, failover, and runbooks. +- Expand API docs and sample payloads. \ No newline at end of file