colin
/
gosint-sitecrawl
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

docs: update roadmap to post v0.0.2; set next goals

This commit is contained in:
colin 2025-08-31 11:14:52 -04:00
parent 95041372e6
commit f4b2fab683
1 changed files with 3 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
TODO.md
View File

@ -1,8 +1,8 @@
## Roadmap (post v0.0.1) ## Roadmap (post v0.0.2)
Prioritized from easiest/low-risk to more involved work. Check off as we ship. Prioritized from easiest/low-risk to more involved work. Check off as we ship.
### Quick wins (target v0.0.2) ### Shipped in v0.0.2
- [x] Add crawl metadata (startedAt, finishedAt, durationMs) - [x] Add crawl metadata (startedAt, finishedAt, durationMs)
- [x] Include run parameters in report (maxDepth, concurrency, timeout, userAgent, sameHostOnly) - [x] Include run parameters in report (maxDepth, concurrency, timeout, userAgent, sameHostOnly)
- [x] Status histogram (2xx/3xx/4xx/5xx totals) in summary - [x] Status histogram (2xx/3xx/4xx/5xx totals) in summary
@ -10,8 +10,6 @@ Prioritized from easiest/low-risk to more involved work. Check off as we ship.
- [x] Add compact `reportSummary` text block to JSON - [x] Add compact `reportSummary` text block to JSON
- [x] Top external domains with counts - [x] Top external domains with counts
- [x] Broken links sample (first N) + per-domain broken counts - [x] Broken links sample (first N) + per-domain broken counts
### Core additions (default, no flags)
- [x] Robots.txt summary (present, fetchedAt) - [x] Robots.txt summary (present, fetchedAt)
- [x] Sitemap extras (index → child sitemaps, fetch errors) - [x] Sitemap extras (index → child sitemaps, fetch errors)
- [x] Per-page response time (responseTimeMs) and content length (basic) - [x] Per-page response time (responseTimeMs) and content length (basic)
@ -19,7 +17,7 @@ Prioritized from easiest/low-risk to more involved work. Check off as we ship.
- [x] Depth distribution (count of pages by depth) - [x] Depth distribution (count of pages by depth)
- [x] Redirect map summary (from → to domain counts) - [x] Redirect map summary (from → to domain counts)
### Outputs and UX ### Next (target v0.0.3)
- [ ] CSV exports: pages.csv, links.csv - [ ] CSV exports: pages.csv, links.csv
- [ ] NDJSON export option for streaming pipelines - [ ] NDJSON export option for streaming pipelines