Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How do I properly use this? #132

Open
TheFantasticLoki opened this issue Jun 21, 2023 · 4 comments
Open

How do I properly use this? #132

TheFantasticLoki opened this issue Jun 21, 2023 · 4 comments

Comments

@TheFantasticLoki
Copy link

I've been trying to get this working for a good 2 hours now and I can't seem to get serve to work. At first I couldn't run proxybroker at all on windows due to aiodns being depreciated among other issues so I pulled the docker container. After about 45 minutes I was able to get it running and working with the find command but when I launch a serve instance withdocker run --rm -p 8888:8888 bluet/proxybroker2 --log CRITICAL serve --host 127.0.0.1 --port 8888 --types HTTP HTTPS --lvl High --min-queue 5 I at least got it running but then when I tried to see if it was working by going to localhost:8888 i got a 400 error after changing the host to 0.0.0.0 so I figured it was working but when I add it to my proxy settings and check my ip I got a different ip ONCE and then it went back to my normal ip but the console shows no connections so idek if it's getting anything.
What do I need to do to get this working properly?

@TheFantasticLoki
Copy link
Author

also even on docker I get these messy errors on the output:
/usr/local/lib/python3.9/asyncio/events.py:80: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10. self._context.run(self._callback, *self._args) Server started at http://0.0.0.0:8888 /usr/local/lib/python3.9/asyncio/trsock.py:20: DeprecationWarning: Using send() method on sockets returned from get_extra_info('socket') will be prohibited in asyncio 3.9. Please report your use case to bugs.python.org. warnings.warn( DeprecationWarning: Enable tracemalloc to get the object allocation traceback /usr/local/lib/python3.9/asyncio/trsock.py:20: DeprecationWarning: Using recv() method on sockets returned from get_extra_info('socket') will be prohibited in asyncio 3.9. Please report your use case to bugs.python.org. warnings.warn( DeprecationWarning: Enable tracemalloc to get the object allocation traceback /usr/local/lib/python3.9/asyncio/trsock.py:20: DeprecationWarning: Using close() method on sockets returned from get_extra_info('socket') will be prohibited in asyncio 3.9. Please report your use case to bugs.python.org. warnings.warn( DeprecationWarning: Enable tracemalloc to get the object allocation traceback /app/proxybroker/server.py:303: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10. await asyncio.gather(*stream, loop=self._loop)

@TheFantasticLoki
Copy link
Author

BTW I want to add some additional info on my process. At first I was just trying to load pages in my browser with the proxy set to the server but after searching through all the issues I saw someone using curl with the proxy flag so I used that for testing afterwords.
I used curl https://ipinfo.io/ip --proxy http://localip:port and until I set the host as 0.0.0.0 curl's proxy connection would abort every time. afterwords I finally got an ip readout but it showed my actual ip instead of a proxy ip. Then I tested in my browser and was able to get a single ip change before it went back to my actual ip again. I never got any output on the proxy server itself saying a connection came through.

@TheFantasticLoki
Copy link
Author

TheFantasticLoki commented Jun 22, 2023

Did some more testing, went ahead and made a dev container using vscode using a python3 template with python 3.8 installed. I still get the same depreciation warnings as before and I still can't get serve to show any connections at all and when I run a curl I still get my own ip nearly every time.

I also went ahead and tried to make a script in order to grab and CHECK proxies and put them into a file so I could try to use proxy.py in order to host a proxy server since this isn't working no matter what I do but I am getting errors saying:
Traceback (most recent call last): File "start.py", line 33, in <module> loop.run_until_complete(grab_and_check_proxies()) File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "start.py", line 13, in grab_and_check_proxies proxy = await broker.get() AttributeError: 'Broker' object has no attribute 'get'
when I run the following code:

import asyncio
import subprocess
from proxybroker import Broker

async def grab_and_check_proxies():
    # Create a new broker instance
    broker = Broker(max_tries=1)

    # Find and check 100 US proxies
    proxies = []
    while len(proxies) < 100:
        await broker.find(types=['HTTP', 'HTTPS'], countries=['US'])
        proxy = await broker.get()
        if proxy is None:
            break
        if proxy.ping < 500:
            proxies.append(f"{proxy.host}:{proxy.port}")
    
    # Write proxies to file
    with open("/workspaces/python/proxy.py/proxies.txt", "w") as f:
        f.write('\n'.join(proxies))

    # Run proxy.py with arguments
    ip = "0.0.0.0"
    subprocess.run(["python", "proxy.py", 
                    "--listen-host", ip, 
                    "--listen-port", "8888",
                    "--upstream-file", "/workspaces/python/proxy.py/proxies.txt"])
                    

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(grab_and_check_proxies())

@TheFantasticLoki
Copy link
Author

I've done some more testing and experimenting, so far I've gotten proxy.py working-ish using the following start script:

#!/bin/bash

# Function to check if a proxy is working
check_proxy() {
  local ip="$1"
  local port="$2"

  if curl --proxy "$ip:$port" -sSfI --max-time 1 "https://ipinfo.io/json" >/dev/null; then
    echo -e "\033[32mWorking proxy\033[0m: $ip:$port"
    return 0
  else
    echo -e "\033[31mNon-working proxy\033[0m: $ip:$port"
    return 1
  fi
}

# Clear contents of working proxies file and proxy pool
working_proxies_file="./working_proxies.txt"

echo "" > "$working_proxies_file"
unset proxy_pool

# Grab US proxies and save to file
proxybroker find --types HTTP HTTPS --lvl High --countries US --strict -l 25 | awk '/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+:[0-9]+/{match($0,/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+:[0-9]+/); print substr($0,RSTART,RLENGTH)}' > ips.txt

while read -r line <&3; do
  ip=$(echo "$line" | cut -d':' -f1)
  port=$(echo "$line" | cut -d':' -f2)

  if check_proxy "$ip" "$port"; then
    echo "$line" >> "$working_proxies_file"
    ips+="$line "
    proxy_pool+=" --proxy-pool $line"
  fi
done 3< "ips.txt"

# Run final command
command="proxy --disable-headers via,X-Forwarded-For,Remote_Addr,X-Real-IP --port 8899 --plugins proxy.plugin.ProxyPoolPlugin$proxy_pool"
echo $command
eval "$command"

Basically the script uses ProxyBroker to find and check proxies then scrape the output IP:PORT Combos to a file called ips.txt which is then read and ran through a curl check to make sure the proxies are indeed working. Then adds each working proxy into a proxy pool for proxy.py. Unfortunately with this setup my ip is always returned even after disabling the headers Via, X-Forwarded-For, Remote_Addr, and X-Real-IP. This is kinda odd since when using proxybroker serve I had at least a 5% chance of a different IP being returned but when using proxy.py I get a 100% return rate of my own IP. I also "tried" to make a plugin for proxy.py that would instead modify the X-Forwarded-For header to replace the ip of the client with a random upstream proxy ip but failed to get proxy.py to load the plugin cause I'm not that smart... I wouldn't mind switching back to just ProxyBroker if anyone can help me get it working properly but all my testing leads me to believe that proxybroker is just broken atm since it doesn't even show incoming connections whereas proxy.py at the very least does that. Please help....

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant