Libgen-text / Текстовая версия книг из Library Genesis

     515766,a3a39bc8254197c861a0cd6f23e4b598,English,pdf,Neal Stephenson,In the Beginning...was the Command Line
     id     md5                              lang    ext author          title

     1501694,43C496B3F2416BE0CE1C3EC52549B7D2,English,epub,"Liu, Cixin",The Three-Body Problem,Three-Body 1
     id      md5                              lang    ext  author       title                  series

    +---------------+----------------------------------+-----------------------+
    | набор данных  | размер                           | количество файлов     |
    |               +----------+--------+--------------+-----------+-----------+
    |               | оригинал | текст  | сжатый текст | оригинал  | текст     |
    +---------------+----------+--------+------------- +-----------+-----------+
    | lg (sci-tech) | 32 ТБ    | 0,8 ТБ | 0,32 ТБ      | 2,450,000 | 2,220,000 |
    +---------------+----------+--------+--------------+-----------+-----------+
    | ff (fiction)  | 2,4 ТБ   | 0,5 ТБ | 0,18 ТБ      | 2,180,000 | 1,890,000 |
    +---------------+----------+--------+--------------+-----------+-----------+

dbs/lg/libgen_compact_2020-04-05.rar                Дамп базы данных lg, скачанный с gen.lib.rus.ec.
                                                    Идентификатор базы данных #4000-4999 находится в файле text/lg/4000.tar.xz и т.д.
dbs/lg/topics.csv.xz                                Тот же дамп базы данных, преобразованный в формат csv
dbs/lg/updated.csv.xz
dbs/lg/simple.csv                                   Упрощенный формат CSV. Отсортирован по языку, автору, названию, расширению
dbs/ff/fiction_2020-04-05.rar                       Дамп базы данных ff, скачанный с gen.lib.rus.ec.
                                                    Идентификатор базы данных #4000-4999 находится в файле text/ff/4000.tar.xz и т.д.
dbs/ff/fiction.csv.xz                               Тот же дамп базы данных, преобразованный в формат CSV
dbs/ff/fiction_description.csv.xz
dbs/ff/simple.csv                                   Упрощенный формат CSV. Отсортирован по языку, автору, названию, расширению
text/lg/0.tar.xz                                    До 1000 преобразованных файлов .txt в каждом архиве .tar.xz
text/lg/1000.tar.xz
...
text/lg/727000.tar.xz
  727000/12324edf5f13c603f05532adc08b7572.pdf.txt   Пример книги внутри архива .tar.xz: текст извлечен из файла
                                                    727000/12324edf5f13c603f05532adc08b7572, полученного из оригинальной lg torrent раздачи
...
text/lg/2454000.tar.xz
text/ff/0.tar.xz                                    До 1000 преобразованных файлов .txt в каждом архиве .tar.xz
text/ff/1000.tar.xz
...
text/ff/727000.tar.xz
    727000/0037b70cf3267a5708eb5f43cca31730.txt     Пример книги внутри архива .tar.xz: текст извлечен из файла
                                                 727000/0037b70cf3267a5708eb5f43cca31730.epub полученного из оригинальной ff torrent раздачи
...
text/ff/2182000.tar.xz

libgen-text / updated 2020-04-05
this is .txt versions of 4 million books from the library genesis collection (the original is 35TB, this is 70x smaller). a .csv library catalog of books is included.
extracting text does not work for all books. prose (paragraphs of text) is usually readable. mathematics texts with equations, or scientific books with graphs, are not readable.
note: '.tar.xz' is a compression format like zip, but 2.5x smaller
      .tar.xz can be extracted by '7zip' in windows, 'the unarchiver' in os x, or 'tar xf' in linux.
===
| how to find a book
===
0. are you on linux? if so, run 'bash programs/find-linux.bash <title or author>'. all books matching your search will be put in a folder named '<title or author>'.
   by default only english books are selected. edit the file to select another language.
1. is it fiction (ff), nonfiction book (lg), or a scientific/academic article (sm)?
   if fiction, open dbs/ff/simple.csv
   if nonfiction, open dbs/lg/simple.csv
   if scientific article, it is not included.
2. search csv by book title or book author.
   search the csv file for the title or author using your favorite tool (excel, text editor etc). authors may be listed by family name, given name, or both.
   make sure the file is in the language you want.
nonfiction (lg) example
  csv search result
     515766,a3a39bc8254197c861a0cd6f23e4b598,English,pdf,Neal Stephenson,In the Beginning...was the Command Line
     id     md5                              lang    ext author          title
3. look at the id and change the last 3 digits to 000. for example here, 515766 -> 515000
4. open text/lg/515000.tar.xz
   inside is a file 515000/a3a39bc8254197c861a0cd6f23e4b598.pdf.txt. this is the book you want. if the book is missing (10%), that means the text could not be extracted.
fiction (ff) example
  csv search result
     1501694,43C496B3F2416BE0CE1C3EC52549B7D2,English,epub,"Liu, Cixin",The Three-Body Problem,Three-Body 1
     id      md5                              lang    ext  author       title                  series
3. look at the id and change the last 3 digits to 000. for example here, 1501694 -> 1501000
4. open text/ff/1501000.tar.xz
   inside is a file 1501000/43c496b3f2416be0ce1c3ec52549b7d2.txt. this is the book you want. if the book is missing (15%), that means the text could not be extracted.
===
| why
===
everyone who can afford a portable hard drive, can have a copy of all books in the world. it doesn't matter whether libgen goes down, whether the entire internet is down, whether your government cuts all the cables out of the country and burns libraries, whether publishing companies start charging a yearly subscription fee. you will already have all books ever written, forever.
===
| why plain text?
===
- text is smaller.
    +----------+------------------------------------+------------------------+
    |          | size                               | number of files        |
    +----------+----------+-------+-----------------+-----------+------------+
    | data set | original | text  | compressed text | original  | text       |
    +----------+----------+-------+-----------------+-----------+------------+
    | lg       | 32TB     | 0.8TB | 0.32TB          | 2,450,000 | 2,220,000  |
    +----------+----------+-------+-----------------+-----------+------------+
    | ff       | 2.4TB    | 0.5TB | 0.18TB          | 2,180,000 | 1,890,000  |
    +----------+----------+-------+-----------------+-----------+------------+
- text can be viewed on every device.
- plain text cannot contain malicious code. pdfs can.
- text can be searched and indexed more easily.
- why csv? anyone can use csv. it can be read raw in a text editor, opened in excel, or imported into a database. database dumps can be used only by experts.
===
| files and organization
===
dbs/lg/libgen_compact_2020-04-05.rar                    database dump of lg downloaded from gen.lib.rus.ec. database id #4000-4999 are found in file text/lg/4000.tar.xz, etc
dbs/lg/topics.csv.xz                                    same database dump converted to csv format
dbs/lg/updated.csv.xz
dbs/lg/simple.csv                                       simplified csv format. sorted by language, author, title, extension
dbs/ff/fiction_2020-04-05.rar                           database dump of ff downloaded from gen.lib.rus.ec. database id #4000-4999 are found in file text/ff/4000.tar.xz, etc
dbs/ff/fiction.csv.xz                                   same database dump converted to csv format
dbs/ff/fiction_description.csv.xz
dbs/ff/simple.csv                                       simplified csv format. sorted by language, author, title, extension
text/lg/0.tar.xz                                        around 1000 converted .txt files in each .tar.xz
text/lg/1000.tar.xz
...
text/lg/727000.tar.xz
    727000/12324edf5f13c603f05532adc08b7572.pdf.txt     example book inside the .tar.xz: the text is extracted from 727000/12324edf5f13c603f05532adc08b7572 in the original lg torrents
...
text/lg/2454000.tar.xz
text/ff/0.tar.xz                                        around 1000 converted .txt files in each .tar.xz
text/ff/1000.tar.xz
...
text/ff/727000.tar.xz
    727000/0037b70cf3267a5708eb5f43cca31730.txt         example book inside the .tar.xz: the text is extracted from 727000/0037b70cf3267a5708eb5f43cca31730.epub in the original ff torrents
...
text/ff/2182000.tar.xz
===
| text extraction
===
text/lg
    file extensions were determined by using the database dumps provided at gen.lib.rus.ec.
    text was extracted using
        .epub: epub2txt
        .pdf: pdftotext
        .djvu: djvutxt
text/ff
    text was extracted using calibre
no ocr was used.
===
| how to help
===
how to make a library of all books available to everyone free forever, even if some people hate it
- give copies to everyone you know
- tell people it's okay to pirate books. it's important to make all knowledge available to everyone, forever.
- don't rely on someone else to keep stuff on the internet for you.
- download a copy of all the torrent files, code, and the latest database dumps. keep it up to date. make it public if possible.
- pick random library genesis torrents and seed them. together we can make it hard to erase or take down.
- scan and ocr some books. upload them. (not that helpful but worth doing)
- translate this readme and distribute your translation, so people speaking other languages can find books
- clean up the entire library genesis collection's metadata (not just a few books)
===
| about me
===
my gpg key is 00BF8D78743D2FCBB70777B85A8C0EB7A8DD275F (available at pgp.mit.edu). you can contact me by encrypted email at choldu8ozpdnm@mail.com.