How to scrape Google Scholar with Python
This guide shows how to scrape Google Scholar with Python: call one endpoint, get every paper as clean JSON with title, authors, year, and citations, then save it to a CSV file. No browser, no HTML parsing.
One request, real response
The hard part of scraping Google Scholar yourself is driving a browser, getting past the block page, and parsing markup that shifts without warning. Instead you send one HTTP request to a search endpoint and get structured results back. First a quick test with curl:
Then the same call in Python with the requests library. Install it with pip install requests, drop in a key, and run. The results live in the organic array:
Here are two of the results that came back, shown as CSV so they drop straight into Sheets or a reference manager:
Scholar returns one page of results per call. To go deeper, add a page number and loop, collecting every result into one list:
Change q to any Scholar query, like graph neural networks or crispr gene editing.
The full record, ready to use
Every result comes complete, so you build a citation list or a research dataset with no second call. Here is one paper as CSV, ready for Sheets or a database:
Or the same record as JSON, exactly as the API returns it:
vs writing your own scraper
You can build a Scholar scraper with a headless browser and a parser, and plenty of tutorials show how. The trade is that Scholar blocks automated traffic quickly, and the layout keeps moving. Here is the difference in practice.
| Writing it yourself | CrustAPI | |
|---|---|---|
| Setup | Install a headless browser, get past the block page and CAPTCHAs | One GET request, no browser |
| Output | Parse the HTML yourself; fields break when the layout changes | Clean JSON from the API, or the same rows as CSV |
| Maintenance | Fix the scraper each time the page changes | We keep it working |
| Free tier | Free, but you pay in time and blocked requests | 3,000 searches / month, no card |
| Cost model | Server, proxy, and developer time | One credit per search, whatever the result count; empty searches free |
| Storing the data | Yours to manage | No storage limits. Export to CSV or a database |
One credit per search
Prepaid packs, and credits never expire: 25k for $49 · 100k for $149 · 500k for $549 · 2.5M for $1,999 · up to 250M, all self-serve, ex-tax. One search costs one credit, however many results come back; a search that returns none costs nothing.
Save the results to a CSV file
Because the response is plain JSON, Python's built-in csv module writes it to a file in a few lines. No extra libraries. This reads every result from the search and writes one row each:
Open papers.csv in Sheets or Excel and you have a finished reading list. Swap the query, run it again, and append to build a bigger literature dataset.
Related: Google Scholar API · Google Search API · Google Patents API · Scrape Google Maps with Python · API docs
When a different tool fits better: Google Scholar has no official API, so there is no formal SLA on this data. This endpoint returns the same public search results anyone sees on scholar.google.com, which is what most citation research and literature-review work needs. For licensed full text or bulk metadata, providers like Crossref and Semantic Scholar run their own APIs.
Common questions
Clay
Try it on a topic you know
3,000 free credits covers a real search. Run one query and look at the JSON that comes back.
Get 3,000 free credits