Lightnovel Crawler for Android

[wruizcanal]

Hi there.

I reviewed the information to deploy the Crawler on Android with Termux and it can be a bit tricky for newbies. I found a simpler way.

All you need is to install this Pydroid app on Android:

https://play.google.com/store/apps/details?id=ru.iiec.pydroid3

Then you create a .py file with this code and save it. Every time you want to use the Crawler you just have to touch the .py file and tell it to open it with Pydroid. It will open it for you and you just have to touch the 'play' button and everything works by itself. You don't have to know anything. This script will look for the Crawler in the Python repositories and install or update it by itself.

Source:

# Script to comfortably run lightnovel-crawler
import os

# Checking Pip version
def update_pip():
	install_update_pip = os.system("pip install --upgrade pip")
	info_pip = ""
	if(install_update_pip == 0):
		os.system("clear")
		return "Pip version check: OK"
	else:
		return "Pip version check %d" % install_update_pip


# Checking Crawler version
def update_crawler():
	install_update_crawler = os.system("pip install --user -U lightnovel-crawler")
	info_crawler = ""
	if(install_update_crawler == 0):
		os.system("clear")
		return "Crawler version check: OK"
	else:
		return "Crawler version check %d" % install_update_crawler


#Start function
def main():
	#Ask if you want to update
	txt = "Do you want to check updates for Crawler? (y/n): "
	if input(txt).lower().strip()[:1] == "y": 
		info_pip = update_pip()
		print(info_pip)
		info_crawler = update_crawler()
		print(info_pip)
		print(info_crawler)

	#Running the Crawler
	help = "To be able to select the options you need to hold down the Cellphone button 'Vol Up' and then press 'W' (Up) or 'S' (Down) to scroll through the options."

	print("============================================================")
	print("ATTENTION")
	print("============================================================")
	print(help)
	print("============================================================")
	
	exc_crawler = os.system("lncrawl")	
	

if __name__ == '__main__':
    main()

I don't know much about Python so the script can be very clumsy. You can modify it to be better, but it works fine.

[dipu-bd]

This might come in handy for new users. I will add a note to README.md linking this issue.

[Durolord]

This didn't work

[shagunchandel]

hello is there a way to crawl lightnovel from android apps instead of website

[shagunchandel]

Its like this i bought premium plus membership of babelnovel which gave me a free book but it's only available on app so I can't unlock the complete chapters from web version so is there a way to download the EPUB for that book from app using my login ?

[dipu-bd]

Oh... I completely misunderstood your question. Sorry, but crawling only works for websites only. It is not possible to crawl from android apps.

[shagunchandel]

Is it possible to develop new program which can help with this ? I just completed my college from mechatronics branch and learning to code while looking for job so can you tell me if it's possible and where should I start like for example in Babelnovel those unlocked books can be downloaded and saved in Android device in encrypted text file is it possible to decrypt it and then convert it into EPUB

[dipu-bd]

I would extract the apk, and look into their code.

[HACKER667]

is there a new version for the pyroid shortcut for ln crawler

[Ivan-Voronyuk]

The latest versions of this packages need grpcio for work. However, I did not manage to install it in termux. (pydroid3 has this package)
After some tests I managed to just download grpcio library for linux-arm64 and just copy it to site-packages folder. This works.
I did not remember the exact way I did this and exact file source, but here is another possible solution (I hope it works):

1. Go here and download file for linux-aarch64 and your python version. Id recommend to use wget -c ` in termux, but browser is also possible.
2. Extract ".conda" file with 7z x <file.conda>
3. Now you get two files "info-grpcio-......tar.zst" and "pkg-grpcio-........tar.zst". Extract pkg file with tar -xf <file>.
4. cp ./lib /data/data/com.termux/files/usr/lib (I hope that lib location in termux is permanent)
5. Now just clean all unnecessary files (*.conda, *.tar.zst, metadata.json and ./lib folder)

If this does not work try installing libgrpc via pkg or apt.

After lncrawler startup you may see Module load failed: ..... grpc._cython ..... error. This does not do anything for me. Everything works as it should.

[Aquaflamber]

Hello i'm new to python and coding in general, i use lncrawler since a few years in .exe on windows, i've try today to get it on android 13 (samsung s20 ultra) i've souvent the first wave of errors but i dont even undestand the second :
a
Downloading wcwidth-0.2.8-py2.py3-none-any.whl.metadata (13 kB)
Collecting protobuf>=3.19.0 (from pyease-grpc>=1.6.0->lightnovel-crawler)
Obtaining dependency information for protobuf>=3.19.0 from https://files.pythonhosted.org/packages/fa/8a/e9c6b48b8f4651df1b1a9d46fe94a74ed99881141b4660aa855a798c7c53/protobuf-4.24.3-py3-none-any.whl.metadata
Downloading protobuf-4.24.3-py3-none-any.whl.metadata (540 bytes)Collecting grpcio (from pyease-grpc>=1.6.0->lightnovel-crawler)
Downloading grpcio-1.59.0.tar.gz (24.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━ 24.8/24.8 MB 21.3 MB/s eta 0:00:00 Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [10 lines of output]
Traceback (most recent call last):
File "", line 2, in
File "", line 34, in
File "/data/data/ru.iiec.pydroid3/cache/pip-install-pu70au8j/grpcio_6e584bf98dd84c679b8add5a6d059bed/setup.py", line 95, in
import commands
File "/data/data/ru.iiec.pydroid3/cache/pip-install-pu70au8j/grpcio_6e584bf98dd84c679b8add5a6d059bed/src/python/grpcio/commands.py", line 31, in
import support
File "/data/data/ru.iiec.pydroid3/cache/pip-install-pu70au8j/grpcio_6e584bf98dd84c679b8add5a6d059bed/src/python/grpcio/support.py", line 111, in
errors.CompileError: diagnose_compile_error,
AttributeError: module 'setuptools.errors' has no attribute 'CompileError'
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
Pip version check: OK
Crawler version check 256

ATTENTION

To be able to select the options you need to hold down the Cellphone button 'Vol Up' and then press 'W' (Up) or 'S' (Down) to scroll through the options.

sh: lncrawl: not found

Please help ^^