Domain skill
macrotrends
Markdown synced from browser-harness domain skills.
- Host
- macrotrends
- Files
- 1
Agent prompt
Use this skill
Copy this prompt into your coding agent to make it enable browser-harness domain skills and read this exact domain folder before automating.
Set up https://github.com/browser-use/browser-harness for me if it is not already installed. If setup is needed, read `install.md` first to install and connect it to my real browser. Then read `SKILL.md` for normal usage and always read `helpers.py` because that is where the browser-harness functions are. Enable domain skills if they are not already enabled by setting `BH_DOMAIN_SKILLS=1` for browser-harness. Use the `macrotrends` domain skill from `agent-workspace/domain-skills/macrotrends/`. Read every markdown file for this domain before inventing an approach: - agent-workspace/domain-skills/macrotrends/scraping.md Use those domain-skill notes to complete my task for `macrotrends` in my real browser. When you open a setup, verification, or task tab, activate it so I can see the active browser tab.
Skill contents
What the agent will read
Data Extraction
scraping.md
- https://www.macrotrends.net — long-term historical financial and economic charts. Three access patterns depending on page type; all work with plain httpget, no browser required.
- All results validated against live site on 2026-04-18.
- Never use the browser for Macrotrends read-only tasks. All endpoints are accessible via httpget with the default Mozilla/5.0 UA. For pages that occasionally 403, switch to a Chrome UA (see gotchas).
- ---
Show full markdown
https://www.macrotrends.net — long-term historical financial and economic charts. Three access patterns depending on page type; all work with plain http_get, no browser required.
All results validated against live site on 2026-04-18.
Do this first: pick your access pattern
| Goal | Pattern | Latency | Variable |
|---|---|---|---|
| Stock OHLCV price history | Direct iframe PHP | ~190ms | dataDaily |
| Stock market cap (daily) | Direct iframe PHP | ~200ms | chartData |
| Stock fundamentals (PE, revenue, margins) | Direct iframe PHP | ~140ms | chartData |
| S&P 500 / composite index charts | chart_iframe_comp.php | ~90ms | originalData |
| Economic indicators (rates, yields, CPI) | /economic-data/ JSON API | ~150ms | data[] array |
| Gold, commodity prices | Either path (both work) | ~150ms | data[] or originalData |
Never use the browser for Macrotrends read-only tasks. All endpoints are accessible via http_get with the default Mozilla/5.0 UA. For pages that occasionally 403, switch to a Chrome UA (see gotchas).
Pattern 1: Stock price history (OHLCV)
Construct the iframe URL directly — no need to fetch the main page first.
import json, re
from helpers import http_get
def get_stock_ohlcv(ticker: str, years_back: int = None) -> list[dict]:
"""
Returns daily OHLCV records for any US stock.
ticker: uppercase ticker symbol, e.g. 'AAPL', 'MSFT', 'TSLA', 'NVDA'
years_back: number of years of history (1=~250 records, 15=~3772 records).
Omit (None) to get ALL available history (AAPL goes back to 1980).
"""
url = f"https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/stock_price_history.php?t={ticker}"
if years_back:
url += f"&yb={years_back}"
html = http_get(url)
m = re.search(r'var\s+dataDaily\s*=\s*\[', html)
if not m:
raise ValueError(f"No dataDaily found for ticker {ticker!r}")
si = html.index('[', m.start())
bc = 0
for j, ch in enumerate(html[si:], si):
if ch == '[': bc += 1
elif ch == ']':
bc -= 1
if bc == 0: ei = j; break
return json.loads(html[si:ei+1])
# Usage
records = get_stock_ohlcv('AAPL', years_back=15)
# [{'d': '2011-04-18', 'o': '9.771', 'h': '9.9547', 'l': '9.593', 'c': '9.9433', 'v': '18.275'}, ...]
latest = records[-1]
# {'d': '2026-04-17', 'o': '266.96', 'h': '272.3', 'l': '266.72', 'c': '270.23',
# 'v': '55.211', 'ma50': '260.554', 'ma200': '251.828'}
print(f"{latest['d']}: close=${latest['c']} vol={latest['v']}M shares")
dataDaily field reference
| Field | Meaning | Type |
|---|---|---|
d | Date (YYYY-MM-DD) | str |
o | Open price (adjusted for splits) | str/float |
h | High | str/float |
l | Low | str/float |
c | Close | str/float |
v | Volume in millions of shares | str/float |
ma50 | 50-day moving average | str/float (appears on recent records only) |
ma200 | 200-day moving average | str/float (appears on recent records only) |
Note: All price values are strings — cast with float(). Volume is millions: 55.211 = 55.2M shares traded.
Confirmed tickers (2026-04-18)
All tested with direct iframe URL, no page fetch needed:
# All work: AAPL, MSFT, TSLA, NVDA, GOOGL, AMZN, META, NFLX, etc.
# 3772 records for yb=15 (goes back to 2011-04-18)
# AAPL full history: 11428 records back to 1980-12-12
Pattern 2: Stock fundamentals (PE ratio, revenue, market cap, margins)
Different PHP files depending on metric. Construct directly.
Market cap (daily, in billions USD)
import json, re
from helpers import http_get
def get_market_cap(ticker: str, years_back: int = 15) -> list[dict]:
url = f"https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/market_cap.php?t={ticker}&yb={years_back}"
html = http_get(url)
m = re.search(r'var\s+chartData\s*=\s*\[', html)
si = html.index('[', m.start())
bc = 0
for j, ch in enumerate(html[si:], si):
if ch == '[': bc += 1
elif ch == ']':
bc -= 1
if bc == 0: ei = j; break
return json.loads(html[si:ei+1])
data = get_market_cap('AAPL')
# [{'date': '2026-04-15', 'v1': 3929.35}, {'date': '2026-04-16', 'v1': 3884.67}, ...]
# v1 = market cap in billions USD
PE ratio, revenue, current ratio (quarterly/annual fundamentals)
import json, re
from helpers import http_get
def get_fundamental(ticker: str, metric_type: str, statement: str,
freq: str = 'Q', years_back: int = 15) -> list[dict]:
"""
freq: 'Q' = quarterly, 'A' = annual
"""
url = (
f"https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/"
f"fundamental_iframe.php?t={ticker}&type={metric_type}&statement={statement}"
f"&freq={freq}&sub=&yb={years_back}"
)
html = http_get(url)
m = re.search(r'var\s+chartData\s*=\s*\[', html)
si = html.index('[', m.start())
bc = 0
for j, ch in enumerate(html[si:], si):
if ch == '[': bc += 1
elif ch == ']':
bc -= 1
if bc == 0: ei = j; break
return json.loads(html[si:ei+1])
# PE ratio
pe = get_fundamental('AAPL', 'pe-ratio', 'price-ratios')
# [{'date': '2025-09-30', 'v1': 254.146, 'v2': 7.46, 'v3': 34.07}, ...]
# v1 = stock price, v2 = quarterly EPS, v3 = PE ratio
# Revenue
rev = get_fundamental('AAPL', 'revenue', 'income-statement')
# [{'date': '2025-12-31', 'v1': 435.617, 'v2': 143.756, 'v3': 15.65}, ...]
# v1 = TTM revenue ($B), v2 = quarterly revenue ($B), v3 = YoY growth %
# Total assets
assets = get_fundamental('AAPL', 'total-assets', 'balance-sheet')
# Current ratio
ratio = get_fundamental('AAPL', 'current-ratio', 'ratios')
Profit margins
def get_profit_margins(ticker: str, years_back: int = 15) -> list[dict]:
url = (
f"https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/"
f"fundamental_metric.php?t={ticker}&chart=profit-margin&sub=&yb={years_back}"
)
html = http_get(url)
m = re.search(r'var\s+chartData\s*=\s*\[', html)
si = html.index('[', m.start())
bc = 0
for j, ch in enumerate(html[si:], si):
if ch == '[': bc += 1
elif ch == ']':
bc -= 1
if bc == 0: ei = j; break
return json.loads(html[si:ei+1])
margins = get_profit_margins('AAPL')
# [{'date': '2025-12-31', 'v1': 47.33, 'v2': 32.38, 'v3': 27.04}, ...]
# v1 = gross margin %, v2 = operating margin %, v3 = net margin %
Dividend yield
def get_dividend_yield(ticker: str, years_back: int = 15) -> list[dict]:
url = f"https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/dividend_yield.php?t={ticker}&yb={years_back}"
html = http_get(url)
m = re.search(r'var\s+chartData\s*=\s*\[', html)
si = html.index('[', m.start())
bc = 0
for j, ch in enumerate(html[si:], si):
if ch == '[': bc += 1
elif ch == ']':
bc -= 1
if bc == 0: ei = j; break
return json.loads(html[si:ei+1])
dy = get_dividend_yield('AAPL')
# [{'date': '2026-04-17', 'c': 270.23, 'ttm_d': 1.03848, 'ttm_dy': 0.3843}, ...]
# c = stock price, ttm_d = TTM dividend ($), ttm_dy = TTM yield (%)
Stock metric URL reference
| Metric | PHP file | Extra params |
|---|---|---|
| Stock price OHLCV | stock_price_history.php | — |
| Market cap (daily) | market_cap.php | — |
| Dividend yield | dividend_yield.php | — |
| Stock splits (price history) | stock_splits.php | — |
| PE ratio | fundamental_iframe.php | type=pe-ratio&statement=price-ratios |
| Revenue | fundamental_iframe.php | type=revenue&statement=income-statement |
| Total assets | fundamental_iframe.php | type=total-assets&statement=balance-sheet |
| Current ratio | fundamental_iframe.php | type=current-ratio&statement=ratios |
| Profit margins | fundamental_metric.php | chart=profit-margin |
Base URL prefix: https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/
All take ?t={TICKER}&yb={N} (or &sub=&yb={N} for the fundamental ones).
Pattern 3: Index and composite charts (S&P 500, Shiller PE, etc.)
These pages embed chart data via chart_iframe_comp.php. The variable is originalData.
import json, re
from helpers import http_get
def extract_index_chart(page_id: int, url_slug: str) -> list[dict]:
"""
page_id: the numeric ID from the page URL, e.g. 2577
url_slug: last segment of the page URL, e.g. 'sp500-pe-ratio-price-to-earnings-chart'
"""
url = f"https://www.macrotrends.net/assets/php/chart_iframe_comp.php?id={page_id}&url={url_slug}"
html = http_get(url)
m = re.search(r'var\s+originalData\s*=\s*\[', html)
if not m:
raise ValueError("originalData not found — this page may use a different pattern")
si = html.index('[', m.start())
bc = 0
for j, ch in enumerate(html[si:], si):
if ch == '[': bc += 1
elif ch == ']':
bc -= 1
if bc == 0: ei = j; break
return json.loads(html[si:ei+1])
# S&P 500 PE ratio (1180 monthly records, 1927-2026)
pe_data = extract_index_chart(2577, 'sp500-pe-ratio-price-to-earnings-chart')
# [{'date': '1927-12-01', 'close': '15.9099'}, ..., {'date': '2026-03-01', 'close': '27.8925'}]
# 'close' is the PE ratio value
# Gold prices (1336 monthly records, 1915-2026)
gold_data = extract_index_chart(1333, 'historical-gold-prices-100-year-chart')
# [{'id': 'GOLDAMGBD228NLBM', 'date': '1915-01-01', 'close': '629.36', 'close1': '19.250'}, ...]
# 'close' = inflation-adjusted price, 'close1' = nominal USD price
print(f"Latest S&P PE: {pe_data[-1]}") # {'date': '2026-03-01', 'close': '27.8925'}
print(f"Latest gold: {gold_data[-1]}") # {'id': ..., 'date': '2026-04-01', 'close': '5177.19', 'close1': '5177.190'}
Detecting which pattern a page uses
def get_page_pattern(page_url: str) -> str:
html = http_get(page_url)
if 'chart_iframe_comp.php' in html:
return 'index_chart' # use extract_index_chart()
elif 'generateChart' in html and 'highchartsURL' in html:
return 'economic_api' # use get_economic_data()
elif '/production/stocks/desktop/PRODUCTION/' in html:
return 'stock_iframe' # use get_stock_ohlcv() etc.
return 'unknown'
To get the ID and slug from a page
import re
from helpers import http_get
page_url = "https://www.macrotrends.net/2577/sp500-pe-ratio-price-to-earnings-chart"
html = http_get(page_url)
# Option A: parse from the iframe src in the HTML
m = re.search(r'chart_iframe_comp\.php\?id=(\d+)&url=([^"&]+)', html)
if m:
page_id, url_slug = int(m.group(1)), m.group(2)
# Option B: derive from the page URL (works when slug matches)
import urllib.parse
parts = page_url.rstrip('/').split('/')
page_id = int(parts[-2]) # 2577
url_slug = parts[-1] # 'sp500-pe-ratio-price-to-earnings-chart'
Pattern 4: Economic indicator API
Pages that use generateChart() in their JS load data from /economic-data/{pageID}/{freq}.
This endpoint requires a Referer header matching the page URL.
import json, datetime, gzip, urllib.request
from helpers import http_get
def get_economic_data(page_id: int, referer_url: str, freq: str = 'D') -> dict:
"""
page_id: numeric ID from the page URL (e.g. 2015 for Fed Funds Rate)
referer_url: the full page URL — required as Referer header
freq: 'D' = daily, 'M' = monthly (not all support both)
Returns {'data': [[ts_ms, value], ...], 'metadata': {...}}
"""
url = f"https://www.macrotrends.net/economic-data/{page_id}/{freq}"
headers = {
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
"Accept": "application/json, */*",
"Accept-Encoding": "gzip",
"Referer": referer_url,
}
with urllib.request.urlopen(urllib.request.Request(url, headers=headers), timeout=20) as r:
raw = r.read()
if r.headers.get("Content-Encoding") == "gzip":
raw = gzip.decompress(raw)
result = json.loads(raw)
if result is None:
raise ValueError(f"pageID={page_id} does not support freq={freq!r}")
return result
# Fed Funds Rate (daily, 25319 records)
ffr = get_economic_data(2015, "https://www.macrotrends.net/2015/fed-funds-rate-historical-chart", freq='D')
print(ffr['metadata']['name']) # 'Fed Funds Interest Rate'
print(ffr['metadata']['label']) # '%'
# Convert timestamps to dates
for ts_ms, value in ffr['data'][-3:]:
dt = datetime.datetime.fromtimestamp(ts_ms / 1000, datetime.UTC)
print(f"{dt.strftime('%Y-%m-%d')}: {value}%")
# 2026-04-13: 3.64%
# 2026-04-14: 3.64%
# 2026-04-15: 3.64%
# 10-Year Treasury yield (daily, 16074 records)
t10 = get_economic_data(2016, "https://www.macrotrends.net/2016/10-year-treasury-bond-rate-yield-chart", freq='D')
# Last: 2026-04-15: 4.29%
# Gold prices (monthly, 1336 records, 1915-present) — template=5
gold = get_economic_data(1333, "https://www.macrotrends.net/1333/historical-gold-prices-100-year-chart", freq='M')
# metadata: {'name': 'Gold Prices', 'currency': '$', 'label': ''}
# US Unemployment Rate (monthly, 938 records)
unemp = get_economic_data(1316, "https://www.macrotrends.net/1316/us-national-unemployment-rate", freq='M')
# metadata: {'name': 'U.S. Unemployment Rate', 'label': '%'}
# Debt-to-GDP ratio (monthly, 712 records)
debt_gdp = get_economic_data(1381, "https://www.macrotrends.net/1381/debt-to-gdp-ratio-historical-chart", freq='M')
metadata fields
{
'name': 'Fed Funds Interest Rate', # chart title
'tableHeaderName': 'Fed Funds Interest Rate',
'currency': '', # '$' for dollar-denominated series
'label': '%', # units label
'chartType': 'line',
'mobileChartType': 'line',
'lineWidth': 2,
'positiveColor': '#2caffe',
'negativeColor': '',
'decimals': '',
'chartScale': 'linear',
'seriesUnits': ''
}
Available frequency codes
| Code | Meaning | Notes |
|---|---|---|
D | Daily | Most series support this |
M | Monthly | Returns null if not available |
Q | Quarterly | Usually null — use M instead |
A | Annual | Usually null — use M instead |
DEFAULT | Default (usually monthly) | Same data as M for most series |
INDEXMONTHLY | Monthly index close | Some commodity/index series |
INDEXDAILY | Daily index | Some series |
DAILYEXCHANGERATE | Daily FX rate | Currency pairs |
10YD | 10-year daily | Specialized series |
Try D first, fall back to M if you get null.
Known economic page IDs
| ID | URL slug | Description |
|---|---|---|
| 1316 | us-national-unemployment-rate | U.S. Unemployment Rate (monthly, back to 1948) |
| 1333 | historical-gold-prices-100-year-chart | Gold Prices (monthly, back to 1915) |
| 1381 | debt-to-gdp-ratio-historical-chart | U.S. Debt to GDP Ratio |
| 2015 | fed-funds-rate-historical-chart | Fed Funds Interest Rate (daily, back to 1954) |
| 2016 | 10-year-treasury-bond-rate-yield-chart | 10-Year Treasury Yield (daily, back to 1962) |
| 2577 | sp500-pe-ratio-price-to-earnings-chart | S&P 500 PE Ratio (uses chart_iframe_comp.php) |
Generic extraction helper
One function that handles all three embedded-JS patterns:
import json, re
from helpers import http_get
def extract_chart_var(html: str, var_name: str) -> list:
"""Extract a JS array variable from Macrotrends iframe HTML."""
m = re.search(rf'var\s+{re.escape(var_name)}\s*=\s*\[', html)
if not m:
return []
si = html.index('[', m.start())
bc = 0
for j, ch in enumerate(html[si:], si):
if ch == '[': bc += 1
elif ch == ']':
bc -= 1
if bc == 0:
return json.loads(html[si:j+1])
return []
# Works for dataDaily, chartData, or originalData:
html = http_get("https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/stock_price_history.php?t=AAPL&yb=15")
daily = extract_chart_var(html, 'dataDaily')
html2 = http_get("https://www.macrotrends.net/assets/php/chart_iframe_comp.php?id=2577&url=sp500-pe-ratio-price-to-earnings-chart")
pe_data = extract_chart_var(html2, 'originalData')
URL construction guide
Stock pages
STOCK_BASE = "https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/"
# Price history OHLCV
f"{STOCK_BASE}stock_price_history.php?t={ticker}" # all history
f"{STOCK_BASE}stock_price_history.php?t={ticker}&yb={years}" # last N years
# Market cap
f"{STOCK_BASE}market_cap.php?t={ticker}&yb={years}"
# Fundamentals
f"{STOCK_BASE}fundamental_iframe.php?t={ticker}&type={type}&statement={stmt}&freq={freq}&sub=&yb={years}"
# type/statement combos: pe-ratio/price-ratios, revenue/income-statement,
# total-assets/balance-sheet, current-ratio/ratios
# Metrics
f"{STOCK_BASE}fundamental_metric.php?t={ticker}&chart={metric}&sub=&yb={years}"
# metrics: profit-margin
# Dividend yield
f"{STOCK_BASE}dividend_yield.php?t={ticker}&yb={years}"
Economic / index pages
# From numeric ID + URL slug (read from page source or page URL)
f"https://www.macrotrends.net/assets/php/chart_iframe_comp.php?id={id}&url={slug}"
# Economic indicator JSON API (requires Referer header)
f"https://www.macrotrends.net/economic-data/{page_id}/{freq}"
Rate limits and anti-bot
- No rate limiting observed at any tested volume. 10 rapid requests to the same stock iframe completed in 1.8s with no throttling, CAPTCHA, or 429 errors.
- Default UA works (
Mozilla/5.0) for most endpoints. The iframe PHP files never 403'd. - Chrome UA needed for some main HTML pages (not data endpoints): use when fetching
/stocks/charts/...or/2015/...wrapper pages if you get 403. Switch to:pythonheaders = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"} - Referer required for
/economic-data/{id}/{freq}— send the page URL asReferer. Without it, the request is allowed but you get a 403 on some pages. - No cookies, sessions, or auth tokens needed for any endpoint.
Gotchas
Main page URL ≠ data page: Some URLs redirect to different content. /1316/us-national-debt-by-year redirects to /1316/us-national-unemployment-rate. Always check the final URL with r.url if the returned data looks wrong. Use the final URL as the Referer.
yb parameter controls history depth:
yb=1→ ~250 records (last year)yb=15→ ~3772 records (last 15 years)- omit → full history (AAPL: 11428 records to 1980; default for most queries)
Two iframe patterns for economic pages: Pages at macrotrends.net/NNNN/slug use either chart_iframe_comp.php (→ originalData) or generateChart + /economic-data/ API. Check the main page HTML to detect which:
if 'chart_iframe_comp.php' in html: # use extract_index_chart()
elif 'highchartsURL' in html: # use get_economic_data()
Gold data has two price columns:
{'id': 'GOLDAMGBD228NLBM', 'date': '2026-04-01', 'close': '5177.19', 'close1': '5177.190'}
# 'close' = inflation-adjusted price (base year adjusts over time)
# 'close1' = nominal USD price (the raw market price)
Economic API frequency codes: Only D and M consistently return data across most series. A and Q return null for most economic indicators. Always try D first.
chartData fields vary by metric:
market_cap.php→{'date', 'v1'}(v1 = market cap in $B)fundamental_iframe.phptype=pe-ratio →{'date', 'v1', 'v2', 'v3'}(stock price, EPS, PE)fundamental_iframe.phptype=revenue →{'date', 'v1', 'v2', 'v3'}(TTM revenue, quarterly revenue, YoY%)fundamental_metric.phpchart=profit-margin →{'date', 'v1', 'v2', 'v3'}(gross%, operating%, net%)dividend_yield.php→{'date', 'c', 'ttm_d', 'ttm_dy'}(price, dividend, yield%)
Bracket matching required for large arrays: The var dataDaily = [...] in stock iframes is ~450KB with 3772 OHLCV records. The re.DOTALL greedy approach works but is slow; bracket-counting (bc pattern above) is O(n) and fast.
No public API for ticker lookup: To find the company slug for a URL, check the search endpoint: https://www.macrotrends.net/production/stocks/desktop/PRODUCTION/ticker_search_list.php?v=YYYYMMDD — but the stock price iframe only needs the ticker symbol (?t=AAPL), not the slug.