diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md new file mode 100644 index 0000000..bdf9def --- /dev/null +++ b/DEVELOPMENT.md @@ -0,0 +1,344 @@ +# Canadian Repair RSS Feed Monitor - Development Guide + +**For Developers:** Technical documentation for maintaining and extending the RSS feed generation system. + +## ๐Ÿ—๏ธ Project Architecture + +### Directory Structure +``` +rss-feedmonitor/ +โ”œโ”€โ”€ README.md # User-facing documentation +โ”œโ”€โ”€ DEVELOPMENT.md # This technical guide +โ”œโ”€โ”€ .gitignore # Git ignore rules +โ”œโ”€โ”€ docs/ # User documentation +โ”‚ โ”œโ”€โ”€ WORKFLOW.md +โ”‚ โ”œโ”€โ”€ QUICK_CREATE_GUIDE.md +โ”‚ โ”œโ”€โ”€ KEYWORD_OPTIMIZATION.md +โ”‚ โ”œโ”€โ”€ canadian-subreddits.md +โ”‚ โ””โ”€โ”€ canadian-repair-searches.md +โ”œโ”€โ”€ scripts/ # Python generation scripts +โ”‚ โ”œโ”€โ”€ generate_modular_rss_feeds.py +โ”‚ โ”œโ”€โ”€ generate_optimized_rss_feeds.py +โ”‚ โ”œโ”€โ”€ generate_practical_rss_feeds.py +โ”‚ โ”œโ”€โ”€ generate_rss_feeds.py +โ”‚ โ”œโ”€โ”€ extract_website_keywords.py +โ”‚ โ””โ”€โ”€ update_keywords_from_website.py +โ”œโ”€โ”€ data/ # Source data files +โ”‚ โ”œโ”€โ”€ repair_keywords.json +โ”‚ โ”œโ”€โ”€ canadian_subreddits.json +โ”‚ โ””โ”€โ”€ remaining_queries.txt +โ”œโ”€โ”€ feeds/ # Generated RSS feeds +โ”‚ โ”œโ”€โ”€ rss-feeds.json +โ”‚ โ”œโ”€โ”€ *.md (generated RSS docs) +โ”‚ โ””โ”€โ”€ *.opml (RSS reader imports) +โ””โ”€โ”€ archive/ # Old/deprecated files + โ”œโ”€โ”€ ALERT_CREATION_PROCESS.md + โ””โ”€โ”€ ALL_SEARCH_LINKS_COMPLETE.txt +``` + +## ๐Ÿ”ง Development Setup + +### Prerequisites +- Python 3.8+ +- No external dependencies for core functionality +- Optional: `pyyaml` for advanced keyword extraction + +### Installation +```bash +# Clone the repository +git clone +cd rss-feedmonitor + +# No pip installs required for basic functionality +# Optional: pip install pyyaml (for extract_website_keywords.py) +``` + +## ๐Ÿ“Š Data Sources + +### repair_keywords.json +**Purpose:** Defines all repair keyword categories and search terms +**Structure:** +```json +{ + "categories": { + "iphone_repairs": { + "name": "iPhone Repair Requests", + "description": "...", + "devices": ["iPhone", "iPhone 12", ...], + "problems": ["repair", "fix", "broken", ...] + } + }, + "additional_keywords": { + "urgency_indicators": ["emergency", "urgent", ...], + "location_indicators": ["local", "near me", ...] + } +} +``` + +### canadian_subreddits.json +**Purpose:** Defines Canadian subreddits with metadata +**Structure:** +```json +{ + "priorities": { + "critical": { + "subreddits": [ + { + "name": "toronto", + "province": "ON", + "population": "2.9M", + "priority_score": 10 + } + ] + } + } +} +``` + +## ๐Ÿ› ๏ธ RSS Generation Scripts + +### generate_modular_rss_feeds.py (Primary) +**Purpose:** Main RSS feed generation script +**Features:** +- Reads from data/ source files +- Generates both Markdown and OPML outputs +- Modular design for easy maintenance +- Handles keyword categorization automatically + +**Usage:** +```bash +cd scripts +python3 generate_modular_rss_feeds.py +``` + +**Output:** +- `feeds/rss_feeds_[timestamp].md` - Human-readable RSS feed documentation +- `feeds/rss_feeds_[timestamp].opml` - RSS reader import file + +### Keyword Update Scripts + +#### update_keywords_from_website.py +**Purpose:** Manually update keywords from motherboardrepair.ca +**Usage:** +```bash +cd scripts +python3 update_keywords_from_website.py +``` + +#### extract_website_keywords.py +**Purpose:** Extract keywords from website YAML/CSV files (requires pyyaml) +**Usage:** +```bash +pip install pyyaml +cd scripts +python3 extract_website_keywords.py +``` + +## ๐Ÿ”„ RSS Feed Generation Process + +### 1. Keyword Processing +```python +# Load keywords from data/repair_keywords.json +keywords = load_keywords() + +# For each category (iphone_repairs, macbook_repairs, etc.) +for category, data in keywords["categories"].items(): + # Extract devices and problems + devices = data["devices"] + problems = data["problems"] + + # Generate search query: (device1 OR device2) AND (problem1 OR problem2) + search_query = build_search_query(devices, problems) +``` + +### 2. URL Generation +```python +# Reddit search RSS format +base_url = "https://www.reddit.com/r/{}/search.rss?q={}&sort=new&type=link" + +# URL encode the search query +encoded_query = urllib.parse.quote(search_query) +rss_url = base_url.format(subreddit_name, encoded_query) +``` + +### 3. Output Generation + +#### Markdown Output +- Hierarchical structure by priority/city/category +- Search queries and RSS URLs for each feed +- Device and problem breakdowns +- Implementation guidance + +#### OPML Output +- XML format for RSS reader bulk import +- Nested outlines by priority/subreddit/category +- RSS XML URLs with proper encoding + +## ๐Ÿ“ Adding New Keywords + +### 1. Edit repair_keywords.json +```json +{ + "categories": { + "new_category": { + "name": "New Device Repairs", + "description": "New device type repair requests", + "devices": ["Device1", "Device2"], + "problems": ["issue1", "issue2", "issue3"] + } + } +} +``` + +### 2. Regenerate RSS Feeds +```bash +cd scripts +python3 generate_modular_rss_feeds.py +``` + +## ๐Ÿ™๏ธ Adding New Canadian Cities + +### 1. Edit canadian_subreddits.json +```json +{ + "priorities": { + "medium": { + "subreddits": [ + { + "name": "newcity", + "province": "AB", + "population": "500K", + "priority_score": 5 + } + ] + } + } +} +``` + +### 2. Regenerate RSS Feeds +```bash +cd scripts +python3 generate_modular_rss_feeds.py +``` + +## ๐Ÿ” Reddit Search RSS Format + +### URL Structure +``` +https://www.reddit.com/r/[subreddit]/search.rss?q=[query]&sort=new&type=link +``` + +### Query Syntax +- **AND operations:** Use `AND` between device and problem groups +- **OR operations:** Use `OR` within device/problem groups +- **Exact phrases:** Use `"quotes"` for multi-word terms +- **URL encoding:** All special characters must be URL-encoded + +### Examples +```python +# iPhone repairs +query = '("iPhone" OR "iPhone 12") AND ("repair" OR "broken")' + +# URL encoded +encoded = urllib.parse.quote(query) +url = f"https://www.reddit.com/r/toronto/search.rss?q={encoded}&sort=new&type=link" +``` + +## ๐Ÿงช Testing RSS Feeds + +### Manual Testing +1. Copy RSS URL to browser +2. Verify feed loads and shows recent posts +3. Check that search results match expected keywords +4. Test OPML import in RSS reader + +### Automated Testing +```bash +# Test feed validity (requires feedparser) +pip install feedparser +python3 -c " +import feedparser +feed = feedparser.parse('YOUR_RSS_URL') +print(f'Feed title: {feed.feed.title}') +print(f'Entries: {len(feed.entries)}') +" +``` + +## ๐Ÿš€ Deployment + +### Git Workflow +```bash +# Update source files +git add data/*.json +git commit -m "Update keywords/subreddits" + +# Regenerate feeds +python3 scripts/generate_modular_rss_feeds.py + +# Commit generated files +git add feeds/ +git commit -m "Regenerate RSS feeds with updated data" + +# Push changes +git push origin main +``` + +### Version Control Strategy +- **Source files** (data/*.json): Always commit changes +- **Generated files** (feeds/*.md, *.opml): Regenerate as needed, commit for distribution +- **Scripts**: Version controlled, update as needed + +## ๐Ÿ› Troubleshooting + +### Common Issues + +**RSS Feed Not Loading:** +- Verify subreddit name is correct +- Check if subreddit restricts RSS access +- Ensure URL encoding is proper + +**No Search Results:** +- Simplify search query (Reddit search has limitations) +- Check keyword spelling and relevance +- Verify subreddit has active repair discussions + +**OPML Import Issues:** +- Validate XML structure +- Check for special characters in URLs +- Test with a single feed first + +### Debug Mode +Add debug prints to scripts: +```python +# In generate_modular_rss_feeds.py +print(f"Processing {len(feeds)} feeds...") +for feed in feeds[:5]: # Debug first 5 + print(f" {feed['subreddit']}: {feed['category_name']}") +``` + +## ๐Ÿ“ˆ Performance Optimization + +### Feed Count Management +- Current: ~322 feeds across 23 subreddits ร— 14 categories +- Monitor RSS reader performance with high feed counts +- Consider priority-based feed generation for large deployments + +### Update Frequency +- **Daily:** Regenerate feeds for latest subreddit activity +- **Weekly:** Update keywords based on lead quality analysis +- **Monthly:** Add new cities/subreddits as market expands + +## ๐Ÿ”’ Security Considerations + +- No API keys or authentication required (Reddit RSS is public) +- Source files contain only public subreddit information +- Generated RSS URLs are safe for public distribution +- No sensitive data stored in repository + +## ๐Ÿ“š Related Documentation + +- `docs/WORKFLOW.md` - User-facing workflow guide +- `docs/QUICK_CREATE_GUIDE.md` - Fast RSS feed creation +- `docs/canadian-repair-searches.md` - Search strategy details +- `README.md` - Project overview for users \ No newline at end of file diff --git a/README.md b/README.md index 99b686f..cf8bdf6 100644 --- a/README.md +++ b/README.md @@ -14,49 +14,41 @@ - **Local Focus**: Users seeking local repair shops and services - **Conversation Tracking**: Subscribe to repair-related thread discussions -## ๐Ÿ“ Files +## ๐Ÿ“š Getting Started -### RSS Generation System โœจ **START HERE** -- **`scripts/generate_modular_rss_feeds.py`** - Generate fresh RSS feeds from source files -- **`data/repair_keywords.json`** - Source file: All repair keywords/keyphrases -- **`data/canadian_subreddits.json`** - Source file: Canadian subreddits with metadata -- **Latest Output:** [feeds/rss_feeds_20260119_141133.md](feeds/rss_feeds_20260119_141133.md) + [OPML](feeds/rss_feeds_20260119_141133.opml) - -### Core Resources -- **`docs/canadian-subreddits.md`** - Complete list of Canadian cities/provinces -- **`docs/canadian-repair-searches.md`** - Keyword search strategies and RSS URL templates -- **`feeds/rss-feeds.json`** - Your active Canadian search RSS feeds +### Quick Setup +1. **Get RSS Feeds:** Download the latest [RSS feeds](feeds/rss_feeds_20260119_141133.md) or [OPML file](feeds/rss_feeds_20260119_141133.opml) +2. **Import to RSS Reader:** Use the OPML file for bulk import into Feedly, Inoreader, etc. +3. **Start Monitoring:** Subscribe to feeds for Toronto, Vancouver, and your target cities +4. **Engage Daily:** Monitor for repair opportunities and respond to relevant posts ### Documentation -- `docs/WORKFLOW.md` - Canadian-first strategy and implementation steps -- `docs/QUICK_CREATE_GUIDE.md` - Fast reference for creating search RSS feeds -- `README.md` - This overview - -### Legacy Files -- `data/remaining_queries.txt` - Old reference (being phased out) +- **[WORKFLOW.md](docs/WORKFLOW.md)** - Step-by-step Canadian-first strategy +- **[QUICK_CREATE_GUIDE.md](docs/QUICK_CREATE_GUIDE.md)** - Fast reference for RSS feeds +- **[canadian-subreddits.md](docs/canadian-subreddits.md)** - Complete Canadian city list +- **[canadian-repair-searches.md](docs/canadian-repair-searches.md)** - Search strategies and examples +- **[DEVELOPMENT.md](DEVELOPMENT.md)** - Technical setup and customization ## ๐Ÿš€ Quick Start -### Canadian-First Setup +### Get Started in 5 Minutes -1. **Choose 2-3 Canadian cities** to start with (see `canadian-subreddits.md`) -2. **Create keyword search RSS feeds** using templates in `canadian-repair-searches.md` -3. **Subscribe to feeds** in your RSS reader -4. **Monitor daily** for repair opportunities -5. **Engage in conversations** of relevant threads +1. **Download the OPML file** from the [latest feeds](feeds/rss_feeds_20260119_141133.opml) +2. **Import into your RSS reader** (Feedly, Inoreader, etc.) +3. **Subscribe to 5-10 feeds** starting with Toronto and Vancouver +4. **Check daily** for repair-related discussions +5. **Engage with potential customers** when you find relevant posts -### Example Search RSS URLs +### What You'll Get -``` -# Toronto iPhone repairs -https://www.reddit.com/r/toronto/search.rss?q=iPhone+repair+OR+"screen+broken"&sort=new&type=link +- **iPhone/MacBook repair requests** in Toronto +- **Laptop/console repair discussions** in Vancouver +- **Screen repairs, charging issues, water damage** posts +- **"Looking for repair shop"** and similar service requests -# Vancouver laptop repairs -https://www.reddit.com/r/vancouver/search.rss?q=laptop+repair+OR+"won't+turn+on"&sort=new&type=link +### Need Custom Feeds? -# Calgary general repairs -https://www.reddit.com/r/calgary/search.rss?q="looking+for+repair"+OR+"need+repair"&sort=new&type=link -``` +For custom cities or different repair categories, see the [DEVELOPMENT.md](DEVELOPMENT.md) guide to generate your own RSS feeds. ## ๐Ÿ™๏ธ Canadian Communities Covered @@ -81,30 +73,11 @@ https://www.reddit.com/r/calgary/search.rss?q="looking+for+repair"+OR+"need+repa - **r/Alberta** - Alberta-wide discussions - **r/Quebec** - Quebec-wide discussions -## ๐Ÿ” RSS Feed Types +## ๐Ÿ” How It Works -### Search RSS Feeds (Primary Method) -Target repair keywords within Canadian subreddits: +**Smart Search RSS Feeds:** We use Reddit's search functionality to find repair-related posts within specific Canadian communities. Instead of monitoring entire subreddits (which would be overwhelming), we target conversations about device repairs, broken screens, charging issues, and service requests. -``` -https://www.reddit.com/r/[subreddit]/search.rss?q=[keywords]&sort=new&type=link -``` - -**Examples:** -- Toronto iPhone repairs: `q=iPhone+repair+OR+"screen+broken"` -- Vancouver laptops: `q=laptop+repair+OR+"won't+turn+on"` -- General repairs: `q="looking+for+repair"+OR+"need+repair"` - -### Standard Subreddit RSS (Supplemental) -Full subreddit feeds (use sparingly to avoid overload): - -``` -https://www.reddit.com/r/[subreddit]/new/.rss -``` - -**Feed options:** -- `new/.rss` - Latest posts -- `hot/.rss` - Popular/trending posts +**Real-Time Updates:** RSS feeds update automatically as new posts appear, giving you instant notifications about potential repair customers in your area. ## Troubleshooting @@ -119,10 +92,10 @@ https://www.reddit.com/r/[subreddit]/new/.rss 2. Monitor multiple related subreddits for better coverage 3. Focus on regional subreddits for local repair opportunities -## Technical Notes +## ๐Ÿ’ก Why This Works -- Reddit RSS feeds update in real-time -- Feeds include post titles, content, and metadata -- No API keys or authentication required -- Feeds are free and publicly accessible -- Monitor feed activity to identify active repair communities +- **Local Focus:** Target Canadian customers seeking local repair services +- **Real-Time:** Get notified instantly when repair discussions happen +- **Free & Easy:** No API keys, accounts, or complex setup required +- **Proven Keywords:** Uses search terms from actual repair businesses +- **Scalable:** Start with 2-3 cities, expand as you handle more leads