How to scrape Google Scholar with Python
One endpoint, every paper as clean JSON, saved to CSV.
How do you scrape Google Scholar?
Scraping Scholar yourself means driving a browser, beating the block page, and parsing markup that shifts. Here you send one HTTP request and read JSON. Three steps:
- 1
Create a free account. Try it free, no card needed.
- 2
Copy your API key. It is on the dashboard right after signup.
- 3
Run the code below. Start with the curl test, then the Python script.
First a quick test with curl:
Then the same call in Python with the requests library. Install it with pip install requests, drop in a key, and run. The results live in the organic array:
Here are two of the results that came back, shown as CSV so they drop straight into Sheets or a reference manager:
Scholar returns one page of results per call. To go deeper, add a page number and loop, collecting every result into one list:
Change q to any Scholar query, like graph neural networks or crispr gene editing.
What data do you get?
Every result comes complete, so a citation list needs no second call. When Scholar lists a free copy, pdfUrl holds a PDF and htmlUrl a web page; both are null when Scholar shows none. Here is one paper as CSV:
Or the same record as JSON, exactly as the API returns it:
vs writing your own scraper
You can build a Scholar scraper with a headless browser and a parser, and plenty of tutorials show how. The trade is that Scholar blocks automated traffic quickly, and the layout keeps moving. Here is the difference in practice.
| Writing it yourself | CrustAPI | |
|---|---|---|
| Setup | Install a headless browser, get past the block page and CAPTCHAs | One GET request, no browser |
| Output | Parse the HTML yourself; fields break when the layout changes | Clean JSON from the API, or the same rows as CSV |
| Maintenance | Fix the scraper each time the page changes | We keep it working |
| Free tier | Free, but you pay in time and blocked requests | $6 of free usage each month |
| Cost model | Server, proxy, and developer time | One charge per search, whatever the result count; empty searches free |
| Storing the data | Yours to manage | No storage limits. Export to CSV or a database |
How much does scraping Google Scholar cost?
Each successful search is billed once at your Google Search rate, however many results come back. Empty searches cost nothing. Paid funds never expire.
| Deposit | Google Search / 1,000 requests | Maps / 1,000 businesses | LinkedIn reads / 1,000 requests | People, Jobs, Refresh / 1,000 units |
|---|---|---|---|---|
| $10+ | $1.00 | $1.96 | $6.00 | $1.96 |
| $149+ | $0.76 | $1.49 | $4.50 | $1.49 |
| $549+ | $0.56 | $1.10 | $3.30 | $1.10 |
| $1,999+ | $0.41 | $0.80 | $2.45 | $0.80 |
| $6,500+ | $0.33 | $0.65 | $1.95 | $0.65 |
| $27,500+ | $0.28 | $0.55 | $1.65 | $0.55 |
| $50,000+ | $0.26 | $0.50 | $1.50 | $0.50 |
| $100,000+ | $0.20 | $0.40 | $1.50 | $0.40 |
Every eligible account gets $6 of free usage monthly. Each deposit keeps its prices until spent; paid funds never expire. People and Jobs bill per successful search, Refresh per accessible profile. Full profiles in People use the read rate per full profile. See all billing details.
Larger deposits unlock lower endpoint prices. Each deposit keeps its prices until spent. See the full deposit table on the pricing page. Prices in USD, ex-tax.
The endpoint's full field list and pricing detail live on the Google Scholar API page.
Save the results to a CSV file
Because the response is plain JSON, Python's built-in csv module writes it to a file in a few lines. No extra libraries. This reads every result from the search and writes one row each:
Open papers.csv in Sheets or Excel and you have a finished reading list. Swap the query, run it again, and append to build a bigger literature dataset.
Who scrapes Google Scholar?
- Researchers. Build a literature dataset for a review or meta-analysis, citation counts included.
- Grad students. Turn a reading list into a CSV with year, venue, and a free-copy link per paper.
- R&D teams. Track how often key papers or a lab's work get cited, on a schedule.
- AI builders. Feed titles, snippets, and PDF links into a research agent or RAG pipeline.
Related: Google Scholar API · Google Search API · Google Patents API · Scrape Google Maps with Python · API docs
When a different tool fits better: Google Scholar has no official API, so there is no formal SLA on this data. This endpoint returns the same public search results anyone sees on scholar.google.com, which is what most citation research and literature-review work needs. For licensed full text or bulk metadata, providers like Crossref and Semantic Scholar run their own APIs.
Common questions
No. With the CrustAPI endpoint you send one HTTP request with the requests library and get clean JSON back. There is no browser to drive and no HTML to parse, so the code above is the whole scraper.
Title, authors and venue, publication year, citation count, a link to the paper, and a direct full-text link when Scholar lists one: pdfUrl when the free copy is a PDF, htmlUrl when it is a web page. Enough to build a citation list or a research dataset without a second call.
Scholar is quick to stop automated traffic, which is why a raw script tends to get blocked fast. The endpoint handles that layer for you, so you just read JSON.
Add a page number to the request and loop, as shown above. Each call returns one page of results, and you pay one charge per search, however many results come back.
The response is plain JSON, so Python's built-in csv module writes it to a file in a few lines. The Save to CSV section above has the exact code, no extra libraries.
Yes. Try it free, no credit card, and paid funds never expire. That is enough to pull several full result pages while you build.
Clay
Try it on a topic you know
the free plan covers a real search. Run one query and look at the JSON that comes back.
Try for free