66 lines
3.6 KiB
Markdown
66 lines
3.6 KiB
Markdown
## Project Ploughshares - OSINT Intake Workflow Demo
|
|
|
|
This demo summarizes the initial implementation of a replacement intake workflow for OSINT signals related to international trade, intended to help detect and deter embargo-evasion patterns. The system focuses on capturing OSINT leads, triaging and approving them, and maintaining an auditable record suitable for collaboration with researchers and analysts.
|
|
|
|
### Goals
|
|
- **Streamline OSINT intake**: Provide a clear path to submit, normalize, review, and approve OSINT signals and related evidence.
|
|
- **Create an auditable workflow**: Every step - from ingestion to analyst approval - is persisted with status and metadata.
|
|
- **Enable automation**: Support ingestion of leads from crawlers and curated feeds to reduce manual toil.
|
|
- **Improve quality**: Bake in accessibility and quality checks (contrast, Lighthouse, basic headers) for the analyst-facing UI.
|
|
|
|
### What we built (repo tour)
|
|
- **Web/API app**: Flask application for UI and JSON endpoints
|
|
- `docker/ploughshares/app.py`, templates in `docker/ploughshares/templates/`, static assets in `docker/ploughshares/static/`
|
|
- Database schema and setup in `docker/ploughshares/schema.sql` and `docker/ploughshares/init_db.py`
|
|
- **Ingestion**: Crawler components to fetch and submit OSINT leads to the API
|
|
- `docker/crawler/*.py` and `docker/crawler_dorks/*.py` (e.g., `write_to_api.py`, `marketline_crawler.py`)
|
|
- **Quality/Accessibility**: Utility scripts and checks
|
|
- `check_accessibility.py`, `check_contrast.py`, `run_lighthouse_test.sh`
|
|
- **Tests and code quality**: `tests/` (API behavior, dependencies, and basic quality gates)
|
|
- **Runtime**: Docker Compose files for local development and deployment stacks
|
|
- `docker-compose.dev.yml`, `docker-compose.yml`, `stack.staging.yml`, `stack.production.yml`
|
|
|
|
### High-level workflow (sequence)
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant Analyst
|
|
participant WebUI as Web UI (Flask)
|
|
participant API as API (Flask/JSON)
|
|
participant DB as Database
|
|
participant Crawler
|
|
|
|
Analyst->>WebUI: Submit OSINT signal for review
|
|
WebUI->>API: POST /transactions (signal + metadata)
|
|
API->>DB: Insert pending transaction with audit fields
|
|
|
|
Crawler->>API: POST /ingest (scraped leads)
|
|
API->>DB: Upsert leads and link evidence
|
|
|
|
Analyst->>WebUI: Review pending approvals
|
|
WebUI->>API: PATCH /transactions/:id approve/deny
|
|
API->>DB: Update status and record decision
|
|
```
|
|
|
|
### Quality gates and accessibility checks (sequence)
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant Dev
|
|
participant CI as CI/Local Checks
|
|
participant LHT as Lighthouse
|
|
participant A11y as Accessibility
|
|
|
|
Dev->>CI: Push or run checks locally
|
|
CI->>LHT: run_lighthouse_test.sh (performance & a11y)
|
|
CI->>A11y: check_accessibility.py / check_contrast.py
|
|
CI->>Dev: Reports and action items
|
|
```
|
|
|
|
### How this supports Project Ploughshares
|
|
- **Mission-aligned intake**: The workflow is designed to help researchers and analysts surface potential embargo-evasion signals in international trade.
|
|
- **Traceability**: Signals and decisions are stored with clear statuses for collaboration and later analysis.
|
|
- **Extensible feeds**: Crawlers and curated feeds can be extended as new sources become relevant.
|
|
|
|
### Next steps (priority first)
|
|
- Implement vector search for intake to perform semantic matching beyond literal keywords and synonyms, surfacing adjacent signals and concepts.
|
|
- Author and test Disaster Recovery Plan (DRP) and Business Continuity Plan (BCP), with RTO/RPO targets, backups, failover, and runbooks.
|
|
- Expand API docs and sample payloads. |