Sentiment Analysis

Sentiment data falls into two broad categories: text-based (qualitative) and market-based (quantitative).

Text-based sentiment uses NLP to extract bullish/bearish signals from written content:

• News headlines and articles — the fastest source of sentiment shifts. A single headline ("FDA rejects XYZ drug application") can move a stock 20-40% before any analyst report is published.
• Social media (Reddit, X/Twitter, StockTwits) — captures retail trader sentiment and early signals that precede mainstream news coverage. The GameStop short squeeze in January 2021 was visible in Reddit sentiment weeks before it hit major financial news.
• Earnings call transcripts — management tone, language complexity, and hedging words ("challenging," "uncertain," "headwinds") correlate with future stock performance. Research shows that more uncertain language on calls predicts lower future returns.
• SEC filings (10-K, 10-Q, 8-K) — subtle changes in risk factor language between filings can signal emerging problems before they become public knowledge.

Market-based sentiment uses trading data to measure how investors are actually positioning:

• Put/call ratio — high ratio (above 1.0) indicates bearish positioning; low ratio (below 0.7) indicates bullish positioning.
• VIX (Volatility Index) — high VIX (above 25) indicates fear; low VIX (below 15) indicates complacency.
• Short interest — high short interest as a percentage of float signals bearish consensus.
• Fund flows — money moving into equity funds signals optimism; flows into bond and money market funds signal caution.

General-purpose sentiment models (like the ones used for product reviews) perform poorly on financial text because financial language has unique semantics. "Cutting" is negative for product quality but can be positive for costs ("cutting expenses"). "Volatile" is neutral in everyday language but has specific meaning in finance.

FinBERT: A BERT model fine-tuned on financial news and reports. It classifies text as positive, negative, or neutral with finance-specific understanding. FinBERT correctly identifies "revenue declined less than expected" as positive sentiment, which generic models often misclassify.

Large Language Models (GPT, Claude): Modern LLMs can perform nuanced financial sentiment analysis including identifying sarcasm, conditional statements ("if the deal closes"), and sentiment toward specific entities within a single article (bullish on the acquirer, bearish on the target). LLM-based sentiment is more accurate but slower and more expensive than FinBERT.

Lexicon-Based Approaches: The simplest method — count positive and negative words from a financial dictionary (like the Loughran-McDonald Financial Sentiment Dictionary). Fast and cheap but misses context, negation ("not profitable"), and complex sentence structures.

For trading applications, the choice depends on latency requirements. For real-time headline scanning, FinBERT or lexicon methods provide sub-second results. For deeper analysis of earnings calls or long-form content, LLMs provide superior accuracy.

Contrarian Approach: Trade against extreme sentiment. When the Fear & Greed Index hits extreme fear (below 20), buy quality stocks that have been sold indiscriminately. When sentiment reaches extreme greed (above 80), reduce exposure and tighten stops. This approach works because sentiment extremes reflect emotional overreaction, and markets tend to revert from extremes. Research shows that buying the S&P 500 during extreme fear readings has historically produced above-average forward returns.

Momentum Approach: Trade with the trend in sentiment. When sentiment turns positive after a period of negativity (e.g., analyst upgrades, positive news flow), buy the momentum. Sentiment momentum works in the short term because positive news attracts more buyers, creating a self-reinforcing cycle. This is particularly effective for individual stock events like earnings surprises and product launches.

Combination: Use market-wide sentiment contrarian signals (buy fear, sell greed) for portfolio-level decisions, and use stock-specific sentiment momentum for individual trade entries. This hybrid approach captures both the reversion of crowd psychology and the persistence of company-specific information flows.

The key insight: sentiment is most useful at extremes. Moderate sentiment levels (neither very bullish nor very bearish) have little predictive value. Focus your sentiment analysis on identifying extremes rather than trying to extract signal from noise in the neutral zone.

Data Sources: Start with one or two reliable sources rather than trying to aggregate everything. Financial news APIs (Finnhub, Benzinga, Alpha Vantage) provide real-time headlines with structured data. Social media APIs provide retail sentiment but require more noise filtering.

Scoring: Normalize all sentiment to a consistent scale (e.g., -1 to +1 or 0 to 100). For NLP sentiment, FinBERT outputs probabilities for positive/negative/neutral — convert these to a single score. For quantitative sentiment (put/call ratio, VIX), use percentile rankings relative to historical data.

Signal Generation: Define thresholds for action. Example: if aggregate sentiment drops below the 10th percentile of its 252-day range, generate a contrarian buy signal. If stock-specific sentiment jumps above the 90th percentile following an event, generate a momentum buy signal.

Integration: Sentiment should be one input among many, not a standalone system. Weight sentiment alongside technical signals, fundamentals, and flow data. A common approach is to use sentiment as a confidence multiplier — a technical setup with confirming sentiment gets a higher conviction score than the same setup with conflicting sentiment.

Backtesting Caution: Sentiment data has significant survivorship bias and look-ahead bias. Historical news data often excludes articles that were later retracted or corrected. Social media data is noisy and platform-dependent (Twitter API changes, Reddit bans). Always test sentiment strategies on out-of-sample data.