How to Stay Ahead: A Software Engineer's Guide to Zotero & arXiv Automation
As a software engineer, staying current with technology is crucial. This tool helps in several key areas
Staying Informed
It keeps you up-to-date with new algorithms, frameworks, and technologies relevant to your work. For example, if your Zotero library is full of papers on machine learning, this tool will recommend new papers in that area.
Saving Time
Instead of manually Browse the arXiv website daily, this tool brings the most relevant papers directly to you. This is a huge time-saver.
Automated Research
It leverages your existing Zotero library to understand your interests. This means the recommendations are personalized and highly relevant, making your research process more efficient.
The tool essentially acts as a personal research assistant, proactively feeding you information tailored to your professional interests.
<br>
Setting up TideDra/zotero-arxiv-daily involves a few straightforward steps. You'll need to use your terminal and a few basic commands.
Clone the Repository
First, get the code onto your local machine.
git clone https://github.com/TideDra/zotero-arxiv-daily.git
cd zotero-arxiv-daily
Install Dependencies
The project relies on Python libraries, so you'll need to install them. It's a good practice to use a virtual environment.
python -m venv venv
source venv/bin/activate # On Windows, use `venv\Scripts\activate`
pip install -r requirements.txt
Configure Zotero
You need to tell the script how to access your Zotero library. This involves getting your Zotero User ID and API key.
Go to Zotero's API Key Generator.
Create a new private key and ensure it has "Read" access.
Copy your User ID and the generated API key.
Create a file named .env in the project directory and add your credentials
ZOTERO_USER_ID=your_user_id
ZOTERO_API_KEY=your_api_key
Run the Script
Now you can run the script to get your daily recommendations.
python main.py
You'll see a list of new, relevant papers printed to your console.
Automate It
To make this truly "daily," you can set up a cron job (on Linux/macOS) or a Task Scheduler (on Windows) to run the script automatically every day. Here's a cron job example
# Open the crontab editor
crontab -e
# Add this line to run the script every day at 9 AM
0 9 * * * /path/to/your/venv/bin/python /path/to/your/zotero-arxiv-daily/main.py
Make sure to replace /path/to/... with your actual file paths.
Let's look at a simplified version of the code to understand its core logic. The script essentially performs three main tasks
Fetches Your Zotero Library
It uses the Zotero API to get the titles and keywords from your existing papers. This is how it learns your interests.
import os
from pyzotero import zotero
# Load credentials from .env
zotero_user_id = os.getenv('ZOTERO_USER_ID')
zotero_api_key = os.getenv('ZOTERO_API_KEY')
zot = zotero.Zotero(zotero_user_id, 'user', zotero_api_key)
items = zot.items()
keywords = set()
for item in items:
# Extract keywords and tags from your library items
if 'tags' in item['data']:
for tag in item['data']['tags']:
keywords.add(tag['tag'])
print(f"Keywords from your library: {keywords}")
Searches arXiv
It then uses these keywords to search the arXiv API for new papers published within the last 24 hours.
import arxiv
# Search arXiv for papers matching your keywords
search_query = " OR ".join(list(keywords)) # e.g., "machine learning OR deep learning"
client = arxiv.Client(page_size=10, delay_seconds=3)
search = arxiv.Search(
query=search_query,
max_results=5,
sort_by=arxiv.SortCriterion.SubmittedDate,
sort_order=arxiv.SortOrder.Descending
)
results = client.results(search)
Filters and Recommends
Finally, it processes the search results, potentially filtering out papers you've already saved, and presents a curated list.
# Simple example of printing results
for paper in results:
print("--------------------")
print(f"Title: {paper.title}")
print(f"Authors: {', '.join([author.name for author in paper.authors])}")
print(f"URL: {paper.pdf_url}")
