Work
Blacksky NLP — Custom Natural Language Processing
What Is Blacksky NLP?
Blacksky NLP is a proprietary, open-source NLP system built as a drop-in replacement for IBM Watson NLU. Rather than paying for enterprise NLP APIs, Blacksky built a modular pipeline using best-in-class open-source models — at zero per-request cost.
It was originally built for the Rap Almanac (Rap Research Labs) but represents a reusable capability Blacksky can deploy for any client that needs text intelligence without vendor lock-in or API costs.
Five Modular Analyzers
1. Sentiment Analysis
- Model: DistilBERT (fine-tuned on SST-2)
- Output: Score (-1.0 to +1.0) + label (positive / negative / neutral)
- Handles long documents by chunking and averaging across segments
2. Emotion Detection
- Model: RoBERTa with GoEmotions fine-tuning (28-label)
- Output: 5-axis scoring — sadness, joy, fear, disgust, anger (each 0–1)
- Watson-compatible output format for easy migration
3. Named Entity Recognition
- Model: spaCy en_core_web_lg
- Extracts: people, places, organizations, events
- Includes false-positive filtering configurable per domain
4. Keyword Extraction
- Model: KeyBERT (all-MiniLM-L6-v2)
- Features: Maximal Marginal Relevance for keyword diversity, 1–3 word phrases
- Returns top 20 keywords with relevance scores
5. Category Classification
- Model: BART-large-mnli (zero-shot)
- Output: Multi-label classification against a custom taxonomy (up to 5 categories per document with confidence scores)
- Taxonomy is fully configurable per client use case
How It Works
The pipeline is fully incremental — it tracks which documents have been processed and only runs new ones. Models are lazy-loaded on first use for fast startup. The full pipeline runs: ingest → sentiment → emotion → entities → keywords → categories → database storage.
Why Blacksky NLP Instead of Watson or OpenAI
- Zero per-request cost — runs on open-source models, no API fees
- No vendor lock-in — you own the models and the pipeline
- Fully customizable — swap models, adjust taxonomies, tune for your domain
- Privacy-first — data never leaves your infrastructure
- Production-grade — incremental processing, error handling, progress logging, and database persistence built in
Use Cases
- Music and media content analysis
- Customer feedback and sentiment monitoring
- Document classification and tagging at scale
- Any application currently using Watson NLU, OpenAI text analysis, or similar paid APIs
Contact
To discuss deploying Blacksky NLP for your use case:
- Email: mario@blackskymedia.org
- Phone: 347.495.2309