[BUG REPORT] - "Too many requests" #4

Activity

1. joe024132

   added

   bugSomething isn't workingSomething isn't working

   on Nov 25, 2024
2. github-actions commented on Nov 25, 2024

   

   github-actionsBot

   on Nov 25, 2024 – with GitHub Actions

   Contributor

   More actions

   Thank you for your first issue. To better understand your request or the problem you've encountered, please provide as many details as possible. If the behavior changes or if you have new information about your request, don't hesitate to add it. It will be reviewed ASAP.
3. SegoCode commented on Nov 26, 2024

   

   SegoCode

   on Nov 26, 2024

   Owner

   More actions

   Hi there,

   Well that happens because you have reached a certain limit from your IP, you can connect to a VPN and continue downloading from there or wait a while. Some solutions would be to implement some “interval” between downloads or to be able to define proxies. Any other ideas?
4. M0ller commented on Nov 27, 2024

   

   M0ller

   on Nov 27, 2024

   More actions

   I got this to and added a interval that paused for a few seconds and then tried again for a couple of times. It works better. But also adding a millisecond wait before each go routine adds a bit more help as well.

   Also did a check to only download the file if it is not already downloaded and if the request succeeds.

   Could add this as a param to the command if it should apply the wait's.
5. SegoCode commented on Nov 27, 2024

   

   SegoCode

   on Nov 27, 2024

   Last edited by SegoCode

   Owner

   More actions

   I got this to and added a interval that paused for a few seconds and then tried again for a couple of times. It works better. But also adding a millisecond wait before each go routine adds a bit more help as well.

   Also did a check to only download the file if it is not already downloaded and if the request succeeds.

   Could add this as a param to the command if it should apply the wait's.

   Yeah, definitely dont download the text “too many request” has to be implemented. And some proxy and interval parameters...

   The problem with adding so many flags is that it becomes unmaintainable, and you would have to use the flag package, the problem arises because of how Go's flag package parses command-line arguments. By default, the flag package stops parsing flags when it encounters the first non-flag argument (in my case, the URL) then, we would always have to define --url, and I think it is already breaking little by little the simplicity...
6. M0ller commented on Nov 28, 2024

   

   M0ller

   on Nov 28, 2024

   More actions

   Yeah true. Maybe a config json file that have some default values for wait timer etc and then later a proxy config or such could be added in it if / when it is developed. That maybe is the way to go?
7. SegoCode commented on Nov 28, 2024

   

   SegoCode

   on Nov 28, 2024

   Last edited by SegoCode

   Owner

   More actions

   I made this snipped, in theory, if the flags are at the beginning they can be parsed until they reach the string URL

   	// Manually parse flags and positional arguments
   	var args []string
   	for i := 1; i < len(os.Args); i++ {
   		arg := os.Args[i]
   		if arg == "--" {
   			// All remaining args are positional
   			args = append(args, os.Args[i+1:]...)
   			break
   		}
   		if strings.HasPrefix(arg, "-") {
   			// Flag
   			fs.Parse(os.Args[i:])
   			break
   		}
   		// Positional argument
   		args = append(args, arg)
   	}
   
   	// After parsing flags, any remaining arguments are positional
   	args = append(args, fs.Args()...)

   I made a branch https://github.com/SegoCode/4cget/tree/develop-flags to test and keep coding this implementation. 888dc23
8. SegoCode commented on Sep 3, 2026

   

   SegoCode

   on Sep 3, 2026

   Owner

   More actions

   Restarting development on this 🎉

   4cget now reads the status first and skips 403/429 instead of calling io.Copy. On those statuses it prints a Cloudflare hint: low IP trust score, browser challenge required. 429 also mentions --sleep.

   Cloudflare scores each GET on boards.4chan.org from IP, ASN, TLS fingerprint, and User-Agent. Normal browser traffic from the same IP raises that score. Open the thread in a browser, browse a bit, then retry the CLI.

   --sleep N serializes downloads with N seconds between files. That is the interval we talked about here. --monitor already waits between thread polls, so the two flags cannot run together.

   I dropped proxy support. 429 on i.4cdn.org is a rate limit on that IP. Spacing requests is the control we have without another hop.

   If 429 shows up again, raise --sleep and retry.
9. SegoCode

   closed this as completedon Sep 3, 2026