Shutdown notice
This website no longer operational due to lack of funds. Contact me (
[email protected]).
Free = Search entire libgen for 30 min (contact me)
$100/mo = Search 10% of libgen. (Will rent 256 GB RAM)
$800/mo = Search entire libgen. (Will rent 2 TB RAM)
$50k = Search entire libgen for lifetime. (Will write code to search using disk not RAM)
Dataset = ~2 TB embeddings ~300M vectors; from ~300 GB plaintext; from ~7 TB ~700k unique english epubs; selected from ~65 TB libgen database
Embedding model = openai text-embedding-3-small
Total budget so far = ~$2000 for servers and openai embedding API
Database and search algo = DragonflyDB
Languages/Frameworks used = perl, bash, nginx, .... mojolicious, jq, htmlq, gnu parallel,
Developer notes: Used bash pipelines in all steps (extracing plaintext from epubs, converting to openai jsonl format, queueing them for openai servers, loading results into DB) to max out disk throughput. No use of nodejs and python in order to avoid memory overflow. OpenAI BatchAPI rate limit documentation is bad, had to figure out some hacks like sending 25 "requests" per batch file, 2048 strings per "request", 20 batch files at a time.