```
├── README.md
├── cmd/
   ├── cf-hero/
      ├── main.go
├── go.mod
├── go.sum
├── img/
   ├── 1.png
   ├── 2.png
   ├── a
   ├── cf-heroo.jpg
   ├── gopher-cf-hero.png
├── internal/
   ├── config/
      ├── config.go
   ├── dns/
      ├── dns.go
   ├── http/
      ├── client.go
   ├── scanner/
      ├── scanner.go
   ├── utils/
      ├── utils.go
├── pkg/
   ├── models/
      ├── models.go
```


## /README.md

## CF-Hero

<h1 align="center">
  CF-Hero
  <br>
</h1>

<p align="center">
  <a href="https://www.blackhat.com/asia-24/arsenal/schedule/#cf-hero-37767"><img src="https://img.shields.io/badge/Black%20Hat-Arsenal%2024-blue"></a>
  <a href="https://pkg.go.dev/github.com/musana/cf-hero"><img src="https://pkg.go.dev/badge/github.com/musana/cf-hero.svg"></a>
  <a href="https://goreportcard.com/report/github.com/musana/cf-hero"><img src="https://goreportcard.com/badge/github.com/musana/cf-hero"></a>
  <a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-green.svg"></a>
  <a href="https://twitter.com/musana"><img src="https://img.shields.io/twitter/follow/musana.svg?logo=twitter"></a>


</p>

<p align="center">
  <a href="#whats-it">What's it?</a> •
  <a href="#feautures">Features</a> •
  <a href="#background">Background</a> •
  <a href="#installation-instructions">Installation</a> •
  <a href="#usage">Usage</a> •
  <a href="#running-cf-hero">Running cf-hero</a> •
  <a href="#to-do">To Do</a>
</p>

<p height="300" align="center">
  <img src="img/gopher-cf-hero.png">
</p>

# What's it?
CF-Hero is a comprehensive reconnaissance tool developed to discover the real IP addresses of web applications protected by Cloudflare. It performs multi-source intelligence gathering through various methods.

### DNS Reconnaissance
- Current DNS records (A, TXT)
- Historical DNS data analysis
- Associated domain discovery

### Intelligence Sources
- Active DNS enumeration
- Censys search engine
- Shodan search engine
- SecurityTrails historical records
- Related domain correlation

The tool analyzes data from these sources to identify potential origin IP addresses of Cloudflare-protected targets. It validates findings through response analysis to minimize false positives.

**a simple flowchart of the tool**

<p align="left">
  <img src="img/cf-heroo.jpg">
</p>

# Feautures
### Features

- DNS Reconnaissance
  - Checks current DNS records (A, TXT)
  - Extracts domains behind Cloudflare
  - Extracts domains not behind Cloudflare

- Third-party Intelligence
  - Censys integration
  - Shodan integration
  - SecurityTrails integration
  - Reverse IP lookup for associated domains

- Advanced Features
  - Custom JA3 fingerprint support
  - Concurrent scanning capabilities
  - Standard input support (piping)
  - HTML title comparison for validation
  - Proxy support
  - Custom User-Agent configuration

# Background
## Current DNS Records
Let's take look at some use-case with misconfigured DNS settings. 

As you can see, a regular DNS query returns the IP address of the domain. For example, musana.net is behind Cloudflare (CF), but sometimes the domain has multiple A records, and some of them may not correspond to IP addresses associated with CF. (This DNS output is merely an illustrative example and may not represent the exact DNS answer for musana.net.)

```
;; ANSWER SECTION:
musana.net.	300	IN	A	104.16.42.102
musana.net.	300	IN	A	104.16.43.102
musana.net.	300	IN	A	123.45.67.89 (Real IP exposed)
musana.net.	300	IN	A	123.45.76.98 (Real IP exposed)
```

The another case is related to TXT records. Sometimes domain is behind of CF but real IP of the domain may used in TXT records. CF-Hero check all TXT records then extract all IP address finally it try to connect IP which it found via HTTP.

Let's say we have like DNS TXT records. As seen in the TXT records, there is SPF record. Some company can host own mail server and TXT records may contain IP which points to target domain.

As you can see in the following DNS answer SPF record has some IP addresses. Cf-Hero also checks these.

```
;; ANSWER SECTION:
musana.net.	115	IN	TXT	"1password-site-verification=LROK6G5XFJG5NF76TE2FBTABUA"
musana.net.	115	IN	TXT	"5fG-7tA-G4V"
musana.net.	115	IN	TXT	"MS=ms16524910"
musana.net.	115	IN	TXT	"OSSRH-74956"
musana.net.	115	IN	TXT	"docker-verification=6910d334-a3fc-419c-89ac-57668af5bf0d"
musana.net.	115	IN	TXT	"docusign=4c6d27bb-572e-4fd4-896c-81bfb0af0aa1"
musana.net.	115	IN	TXT	"shopify-verification-code=1Ww5VsPpkIf32cJ5PdDHdguRk22K2R"
musana.net.	115	IN	TXT	"shopify-verification-code=NM243t2faQbaJs8SRFMSEQAc4J9UQf"
musana.net.	115	IN	TXT	"v=spf1 include:_spf.google.com include:cust-spf.exacttarget.com include:amazonses.com include:mail.zendesk.com include:servers.mcsv.net include:spf.mailjet.com ip4:216.74.162.13 ip4:216.74.162.14  ip4:153.95.95.86 ip4:18.197.36.5 -all"


```

## OSINT

OSINT is another technique to find real IP of any domain which is behind of CF. There are lots of special search engine for special purpose. Shodan and Censys are two of these. They provide more detail and technical information.  These search engine scan whole internet continously and discover new assets or monitor and log changing in assets. When a domain which is not behind of CF get up, bot of these engine can log Real IP of the domain. After a while if the domain will take behind of cloudflare, their IP can be found using these search engine.

CF-Hero checks censys and shodan too. (Note that when you use these services you have some limit due to API quota.)


## (Sub)Domains
The other trick way is (sub)domain technique. Actually It doesn't have to be a subdomain It can be domain as well. The key point is here; domains should belong to same company.

Let's say we have 2 domain. One of them is behind of CF but the otner is not. In this case, you connect to the domain which is not behind of CF then you change host header with domain which is behind of CF. If you get response of application's which is behind of CF, congruculations you bypassed CF. You can access web application from IP directly anymore. (and of course that's also depends on the configuration)


Let's take look at closer


```

--> TCP --> blog.musana.net [123.45.67.89] ---> HTTPs -------------\
                                                                    \
--> TCP --> api.musana.net [123.67.45.98] ----> HTTPs -----------\   \ 
                                                                  \   \
--> TCP --> test.musana.net [123.89.44.88] ---> HTTPs -------------\   \
                                                                    \___\____________________
--> TCP --> tools.musana.net [123.44.55.66] --> HTTPs -------------> | GET / HTTP/2          |
                                                                     | Host: musana.net      | ====> Check & Compare Responses
--> TCP --> admin.musana.net [33.44.123.45] --> HTTPs -------------->|_______________________|
                                                                          /    /
--> TCP --> ... [...] ------------------------> HTTPs ------------------>/    /
                                                                        /    /
--> TCP --> ... [...] ------------------------> HTTPs ---------------->/    /
                                                                      /    /
--> TCP --> random-test.com [55.44.11.33] ----> HTTPs -------------->/    /
                                                                         /
--> TCP --> fsubsidiary.net [66.77.22.123] ---> HTTPs ----------------->/


```


## Historical DNS Records
Historical DNS records services try to discover all domains on the Internet and record changes in the DNS records of these domains.The best known of these services is securitytrails. If a domain is published on internet with its real IP address these service's bot can log it's real IP address after then if the domain is taken behind of cloudflare, the real IP addresses can find out using these services. Thus, we can find the real ip address of a domain that has broadcast over the real ip address in the past.

It uses the security trails service for historical DNS records. You can perform this scan using the `-securitytrails` parameter after entering the API key in the cf-hero.yaml file.


# Installation Instructions

cf-hero requires **go1.18** to install successfully. Run the following command to install.

```
go install -v github.com/musana/cf-hero/cmd/cf-hero@latest
```

# Usage

```

        ____         __
  _____/ __/        / /_  ___  _________
 / ___/ /__  ___   / __ \/ _ \/ ___/ __ \
/ /__/ ___/ (___) / / / /  __/ /  / /_/ /
\___/_/          /_/ /_/\___/_/   \____/

                                @musana
_____________________________________________

Unmask the origin IPs of Cloudflare-protected domains

Usage:
  cf-hero [flags]

Flags:
GENERAL OPTIONS:
   -w int     Worker count (default 16)
   -f string  Input file containing list of host/domain

PRINT OPTIONS:
   -cf      Print domains behind of Cloudflare
   -non-cf  Print domains not behind of Cloudflare

SOURCES:
   -censys          Include Censys in scanning
   -securitytrails  Include SecurityTrails historical DNS records in scanning
   -shodan          Include Shodan historical DNS records in scanning
   -dl string       Domain list for sub/domain scanning
   -td string       Target domain for sub/domain scanning

CONFIGURATION:
   -hm string   HTTP method. (default "GET")
   -ja3 string  JA3 String (default "772,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,18-10-16-23-45-35-5-11-13-65281-0-51-43-17513-27,29-23-24,0")
   -ua string   HTTP User-Agent (default "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/113.0")
   -px string   HTTP proxy URL


```


# Running CF-Hero
The most basic running command. It checks A and TXT records by default.

```
# cat domains.txt | cf-hero
```

or you can pass "f" parameter to it.
```
# cf-hero -f domains.txt
```

Use the **censys** parameter to include Shodan in the scan
```
# cat domain.txt | cf-hero -censys
```

Use the **shodan** parameter to include Shodan in the scan
```
# cat domain.txt | cf-hero -shodan
```

Use the **securitytrails** parameter to include Shodan in the scan
```
# cat domain.txt | cf-hero -securitytrails
```

Use the -td and -dl parameters to attempt to find the target domain's IP address by utilizing a list of domains or subdomains that are not behind Cloudflare. By specifying the IP addresses in the blocks where you have identified live IP addresses used by the target's cloud or on-premises infrastructure with the -dl parameter, you can find the real IP address of the target domain
```
# cf-hero -td https://musana.net -dl sub_domainlist.txt
```

to get domains behind of CF

```
# cf-hero -f domains.txt -cf
```

to get domains not behind of CF

```
# cf-hero -f domains.txt -non-cf
```

other options (custom ja3, proxy, worker, user agent)

```
# cf-hero -d https://musana.net -ua "Mozilla" -w 32 -ja3 "771,22..." -px "http://127.0.0.1:8080"
```

create cf-hero.yaml file under $HOME/.config/ directory to set censys API key
```
# touch ~/.config/cf-hero.yaml

// content of YAML file should be like;

securitytrails:
  - "api_key_here"
shodan:
  - "api_key_here"
censys:
  - "api_key_here"

```

## SS
<p align="left">
  <img src="img/1.png">
</p>

<p align="left">
  <img src="img/2.png">
</p>

## To Do
- JA3 Randomization. (In some cases, Cloudflare blocks the JA3 hash of libraries used for automation/scanning purposes at the TLS layer. This feature is designed to bypass that protection. You can currently provide a custom JA3 string to bypass this protection.)
- A more effective technique will be added to determine if two HTTP responses are the same.


## /cmd/cf-hero/main.go

```go path="/cmd/cf-hero/main.go" 
package main

import (
	"fmt"
	"os"

	"github.com/gammazero/workerpool"
	"github.com/musana/cf-hero/internal/config"
	"github.com/musana/cf-hero/internal/scanner"
	"github.com/musana/cf-hero/internal/utils"
)

func main() {
	fmt.Print(utils.Banner())

	options := config.ParseOptions()
	var urls []string
	var domainList []string

	if options.File != "" && options.DomainList == "" {
		urls = utils.ReadFromFile(options.File)
	} else if options.File == "" && options.DomainList != "" {
		urls = append(urls, options.TargetDomain)
		domainList = utils.ReadFromFile(options.DomainList)
	} else {
		fi, _ := os.Stdin.Stat()
		if fi.Mode()&os.ModeNamedPipe == 0 {
			fmt.Println("[!] No data found in pipe. Urls must be given using pipe or f parameter!")
			os.Exit(1)
		} else {
			urls = utils.ReadFromStdin()
		}
	}

	scanner := scanner.New(options, urls, domainList)
	scanner.PreScan()

	wp := workerpool.New(options.Worker)
	for _, url := range urls {
		url := url
		wp.Submit(func() {
			scanner.Start(url)
		})
	}
	wp.StopWait()
}

```

## /go.mod

```mod path="/go.mod" 
module github.com/musana/cf-hero

go 1.20

require (
	github.com/Danny-Dasilva/CycleTLS/cycletls v0.0.0-20220620102923-c84d740b4757
	github.com/gammazero/workerpool v1.1.3
	github.com/hashicorp/go-retryablehttp v0.7.4
	github.com/miekg/dns v1.1.55
	github.com/projectdiscovery/goflags v0.1.11
	github.com/projectdiscovery/retryabledns v1.0.24
	github.com/schollz/progressbar/v3 v3.13.1
	golang.org/x/net v0.12.0
	gopkg.in/yaml.v2 v2.4.0
)

require (
	github.com/Danny-Dasilva/fhttp v0.0.0-20220524230104-f801520157d6 // indirect
	github.com/Danny-Dasilva/utls v0.0.0-20220604023528-30cb107b834e // indirect
	github.com/Mzack9999/go-http-digest-auth-client v0.6.1-0.20220414142836-eb8883508809 // indirect
	github.com/andybalholm/brotli v1.0.4 // indirect
	github.com/asaskevich/govalidator v0.0.0-20210307081110-f21760c49a8d // indirect
	github.com/aymerick/douceur v0.2.0 // indirect
	github.com/cnf/structhash v0.0.0-20201127153200-e1b16c1ebc08 // indirect
	github.com/dsnet/compress v0.0.1 // indirect
	github.com/gammazero/deque v0.2.0 // indirect
	github.com/gorilla/css v1.0.0 // indirect
	github.com/gorilla/websocket v1.5.0 // indirect
	github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
	github.com/mattn/go-runewidth v0.0.14 // indirect
	github.com/microcosm-cc/bluemonday v1.0.24 // indirect
	github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db // indirect
	github.com/pkg/errors v0.9.1 // indirect
	github.com/projectdiscovery/blackrock v0.0.1 // indirect
	github.com/projectdiscovery/retryablehttp-go v1.0.15 // indirect
	github.com/projectdiscovery/utils v0.0.40-0.20230627061640-8ec2b35f851c // indirect
	github.com/rivo/uniseg v0.4.4 // indirect
	github.com/saintfish/chardet v0.0.0-20120816061221-3af4cd4741ca // indirect
	go.uber.org/multierr v1.11.0 // indirect
	golang.org/x/crypto v0.11.0 // indirect
	golang.org/x/exp v0.0.0-20221019170559-20944726eadf // indirect
	golang.org/x/mod v0.8.0 // indirect
	golang.org/x/sys v0.10.0 // indirect
	golang.org/x/term v0.10.0 // indirect
	golang.org/x/text v0.11.0 // indirect
	golang.org/x/tools v0.6.0 // indirect
	gopkg.in/yaml.v3 v3.0.1 // indirect
)

```

## /go.sum

```sum path="/go.sum" 
github.com/Danny-Dasilva/CycleTLS/cycletls v0.0.0-20220620102923-c84d740b4757 h1:QH22vGS2DO07khPzKe3/CgFevznQkip5WNGEsQX7mFI=
github.com/Danny-Dasilva/CycleTLS/cycletls v0.0.0-20220620102923-c84d740b4757/go.mod h1:R4Hj85bdRH8zqymQ/oZUCmEsODgP3NpUvTEJtaVai7Y=
github.com/Danny-Dasilva/fhttp v0.0.0-20220418170016-5ea1c560e6a8/go.mod h1:t534vrahRNn9ax1tRiYSUvwJSa9jWaYYgETlfodBPm4=
github.com/Danny-Dasilva/fhttp v0.0.0-20220524230104-f801520157d6 h1:Wzbitazy0HugGNRACX7ZB1En21LT/TiVF6YbxoTTqN8=
github.com/Danny-Dasilva/fhttp v0.0.0-20220524230104-f801520157d6/go.mod h1:2IT2IFG+d+zzFuj3+ksGtVytcCBsF402zMNWHsWhD2U=
github.com/Danny-Dasilva/utls v0.0.0-20220418055514-7c61e0dbb504/go.mod h1:A2g8gPTJWDD3Y4iCTNon2vG3VcjdTBcgWBlZtopfNxU=
github.com/Danny-Dasilva/utls v0.0.0-20220418175931-f38e470e04f2/go.mod h1:A2g8gPTJWDD3Y4iCTNon2vG3VcjdTBcgWBlZtopfNxU=
github.com/Danny-Dasilva/utls v0.0.0-20220604023528-30cb107b834e h1:tqiguW0yAcIwQBQtD+d2rjBnboqB7CwG1OZ12F8avX8=
github.com/Danny-Dasilva/utls v0.0.0-20220604023528-30cb107b834e/go.mod h1:ssfbVNUfWJVRfW41RTpedOUlGXSq3J6aLmirUVkDgJk=
github.com/Mzack9999/go-http-digest-auth-client v0.6.1-0.20220414142836-eb8883508809 h1:ZbFL+BDfBqegi+/Ssh7im5+aQfBRx6it+kHnC7jaDU8=
github.com/Mzack9999/go-http-digest-auth-client v0.6.1-0.20220414142836-eb8883508809/go.mod h1:upgc3Zs45jBDnBT4tVRgRcgm26ABpaP7MoTSdgysca4=
github.com/PuerkitoBio/goquery v1.8.0/go.mod h1:ypIiRMtY7COPGk+I/YbZLbxsxn9g5ejnI2HSMtkjZvI=
github.com/andybalholm/brotli v1.0.4 h1:V7DdXeJtZscaqfNuAdSRuRFzuiKlHSC/Zh3zl9qY3JY=
github.com/andybalholm/brotli v1.0.4/go.mod h1:fO7iG3H7G2nSZ7m0zPUDn85XEX2GTukHGRSepvi9Eig=
github.com/andybalholm/cascadia v1.3.1/go.mod h1:R4bJ1UQfqADjvDa4P6HZHLh/3OxWWEqc0Sk8XGwHqvA=
github.com/asaskevich/govalidator v0.0.0-20210307081110-f21760c49a8d h1:Byv0BzEl3/e6D5CLfI0j/7hiIEtvGVFPCZ7Ei2oq8iQ=
github.com/asaskevich/govalidator v0.0.0-20210307081110-f21760c49a8d/go.mod h1:WaHUgvxTVq04UNunO+XhnAqY/wQc+bxr74GqbsZ/Jqw=
github.com/aymerick/douceur v0.2.0 h1:Mv+mAeH1Q+n9Fr+oyamOlAkUNPWPlA8PPGR0QAaYuPk=
github.com/aymerick/douceur v0.2.0/go.mod h1:wlT5vV2O3h55X9m7iVYN0TBM0NH/MmbLnd30/FjWUq4=
github.com/cnf/structhash v0.0.0-20201127153200-e1b16c1ebc08 h1:ox2F0PSMlrAAiAdknSRMDrAr8mfxPCfSZolH+/qQnyQ=
github.com/cnf/structhash v0.0.0-20201127153200-e1b16c1ebc08/go.mod h1:pCxVEbcm3AMg7ejXyorUXi6HQCzOIBf7zEDVPtw0/U4=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dsnet/compress v0.0.1 h1:PlZu0n3Tuv04TzpfPbrnI0HW/YwodEXDS+oPKahKF0Q=
github.com/dsnet/compress v0.0.1/go.mod h1:Aw8dCMJ7RioblQeTqt88akK31OvO8Dhf5JflhBbQEHo=
github.com/dsnet/golib v0.0.0-20171103203638-1ea166775780/go.mod h1:Lj+Z9rebOhdfkVLjJ8T6VcRQv3SXugXy999NBtR9aFY=
github.com/gammazero/deque v0.2.0 h1:SkieyNB4bg2/uZZLxvya0Pq6diUlwx7m2TeT7GAIWaA=
github.com/gammazero/deque v0.2.0/go.mod h1:LFroj8x4cMYCukHJDbxFCkT+r9AndaJnFMuZDV34tuU=
github.com/gammazero/workerpool v1.1.3 h1:WixN4xzukFoN0XSeXF6puqEqFTl2mECI9S6W44HWy9Q=
github.com/gammazero/workerpool v1.1.3/go.mod h1:wPjyBLDbyKnUn2XwwyD3EEwo9dHutia9/fwNmSHWACc=
github.com/gorilla/css v1.0.0 h1:BQqNyPTi50JCFMTw/b67hByjMVXZRwGha6wxVGkeihY=
github.com/gorilla/css v1.0.0/go.mod h1:Dn721qIggHpt4+EFCcTLTU/vk5ySda2ReITrtgBl60c=
github.com/gorilla/websocket v1.5.0 h1:PPwGk2jz7EePpoHN/+ClbZu8SPxiqlu12wZP/3sWmnc=
github.com/gorilla/websocket v1.5.0/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
github.com/hashicorp/go-cleanhttp v0.5.2 h1:035FKYIWjmULyFRBKPs8TBQoi0x6d9G4xc9neXJWAZQ=
github.com/hashicorp/go-cleanhttp v0.5.2/go.mod h1:kO/YDlP8L1346E6Sodw+PrpBSV4/SoxCXGY6BqNFT48=
github.com/hashicorp/go-hclog v0.9.2 h1:CG6TE5H9/JXsFWJCfoIVpKFIkFe6ysEuHirp4DxCsHI=
github.com/hashicorp/go-hclog v0.9.2/go.mod h1:5CU+agLiy3J7N7QjHK5d05KxGsuXiQLrjA0H7acj2lQ=
github.com/hashicorp/go-retryablehttp v0.7.4 h1:ZQgVdpTdAL7WpMIwLzCfbalOcSUdkDZnpUv3/+BxzFA=
github.com/hashicorp/go-retryablehttp v0.7.4/go.mod h1:Jy/gPYAdjqffZ/yFGCFV2doI5wjtH1ewM9u8iYVjtX8=
github.com/k0kubun/go-ansi v0.0.0-20180517002512-3bf9e2903213/go.mod h1:vNUNkEQ1e29fT/6vq2aBdFsgNPmy8qMdSay1npru+Sw=
github.com/klauspost/compress v1.4.1/go.mod h1:RyIbtBH6LamlWaDj8nUwkbUhJ87Yi3uG0guNDohfE1A=
github.com/klauspost/cpuid v1.2.0/go.mod h1:Pj4uuM528wm8OyEC2QMXAi2YiTZ96dNQPGgoMS4s3ek=
github.com/mattn/go-isatty v0.0.17/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM=
github.com/mattn/go-runewidth v0.0.14 h1:+xnbZSEeDbOIg5/mE6JF0w6n9duR1l3/WmbinWVwUuU=
github.com/mattn/go-runewidth v0.0.14/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w=
github.com/microcosm-cc/bluemonday v1.0.24 h1:NGQoPtwGVcbGkKfvyYk1yRqknzBuoMiUrO6R7uFTPlw=
github.com/microcosm-cc/bluemonday v1.0.24/go.mod h1:ArQySAMps0790cHSkdPEJ7bGkF2VePWH773hsJNSHf8=
github.com/miekg/dns v1.1.55 h1:GoQ4hpsj0nFLYe+bWiCToyrBEJXkQfOOIvFGFy0lEgo=
github.com/miekg/dns v1.1.55/go.mod h1:uInx36IzPl7FYnDcMeVWxj9byh7DutNykX4G9Sj60FY=
github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db h1:62I3jR2EmQ4l5rM/4FEfDWcRD+abF5XlKShorW5LRoQ=
github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db/go.mod h1:l0dey0ia/Uv7NcFFVbCLtqEBQbrT4OCwCSKTEv6enCw=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/projectdiscovery/blackrock v0.0.1 h1:lHQqhaaEFjgf5WkuItbpeCZv2DUIE45k0VbGJyft6LQ=
github.com/projectdiscovery/blackrock v0.0.1/go.mod h1:ANUtjDfaVrqB453bzToU+YB4cUbvBRpLvEwoWIwlTss=
github.com/projectdiscovery/goflags v0.1.11 h1:C4UTO3SM5Vfy1J2sdhukm7wONW/tljMpUMNKue5ie00=
github.com/projectdiscovery/goflags v0.1.11/go.mod h1:wC5uJonjddDcCqDNfPq+03nRessSB/LLaaIea4w47ws=
github.com/projectdiscovery/retryabledns v1.0.24 h1:CbC0a1EcyRDBcGFHZDGfW5orkWkOCfa0mAMF060XJpI=
github.com/projectdiscovery/retryabledns v1.0.24/go.mod h1:bCmv0neiqgemgmFChevfX2BgCxIp8sn5OnbwL1Gov9M=
github.com/projectdiscovery/retryablehttp-go v1.0.15 h1:kP9x9f++QimRwb8ABqnI1dhEymvnZXS2Wp2Zs4rWk/c=
github.com/projectdiscovery/retryablehttp-go v1.0.15/go.mod h1:+OzSFUv3sQcPt+MgbNx6X/Q3ESxqPUQSphqG5kxoIgI=
github.com/projectdiscovery/utils v0.0.40-0.20230627061640-8ec2b35f851c h1:mNV/VSMi9wVpq3gcz4km2oUml9M+La20GaFoJPe3Ils=
github.com/projectdiscovery/utils v0.0.40-0.20230627061640-8ec2b35f851c/go.mod h1:rrd8dTBuKEScNMLgs1Xiu8rPCVeR0QTzmRcQ5iM3ymo=
github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
github.com/rivo/uniseg v0.4.4 h1:8TfxU8dW6PdqD27gjM8MVNuicgxIjxpm4K7x4jp8sis=
github.com/rivo/uniseg v0.4.4/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
github.com/saintfish/chardet v0.0.0-20120816061221-3af4cd4741ca h1:NugYot0LIVPxTvN8n+Kvkn6TrbMyxQiuvKdEwFdR9vI=
github.com/saintfish/chardet v0.0.0-20120816061221-3af4cd4741ca/go.mod h1:uugorj2VCxiV1x+LzaIdVa9b4S4qGAcH6cbhh4qVxOU=
github.com/schollz/progressbar/v3 v3.13.1 h1:o8rySDYiQ59Mwzy2FELeHY5ZARXZTVJC7iHD6PEFUiE=
github.com/schollz/progressbar/v3 v3.13.1/go.mod h1:xvrbki8kfT1fzWzBT/UZd9L6GA+jdL7HAgq2RFnO6fQ=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk=
github.com/ulikunitz/xz v0.5.6/go.mod h1:2bypXElzHzzJZwzH67Y6wb67pO62Rzfn7BSiF4ABRW8=
gitlab.com/yawning/bsaes.git v0.0.0-20190805113838-0a714cd429ec/go.mod h1:BZ1RAoRPbCxum9Grlv5aeksu2H8BiKehBYooU2LFiOQ=
go.uber.org/goleak v1.1.12 h1:gZAh5/EyT/HQwlpkCy6wTpqfH9H8Lz8zbm3dZh+OyzA=
go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0=
go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20190325154230-a5d413f7728c/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20220411220226-7b82a4e95df4/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
golang.org/x/crypto v0.0.0-20220518034528-6f7dac969898/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
golang.org/x/crypto v0.0.0-20220525230936-793ad666bf5e/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
golang.org/x/crypto v0.11.0 h1:6Ewdq3tDic1mg5xRO4milcWCfMVQhI4NkqWWvqejpuA=
golang.org/x/crypto v0.11.0/go.mod h1:xgJhtzW8F9jGdVFWZESrid1U1bjeNy4zgy5cRr/CIio=
golang.org/x/exp v0.0.0-20221019170559-20944726eadf h1:nFVjjKDgNY37+ZSYCJmtYf7tOlfQswHqplG2eosjOMg=
golang.org/x/exp v0.0.0-20221019170559-20944726eadf/go.mod h1:cyybsKvd6eL0RnXn6p/Grxp8F5bW7iYuBgsNCOHpMYE=
golang.org/x/mod v0.8.0 h1:LUYupSeNrTNCGzR/hVBk2NHZO4hXcVaW1k4Qx7rjPx8=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/net v0.0.0-20190328230028-74de082e2cca/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20210610132358-84b48f89b13b/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20210916014120-12bc252f5db8/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20220412020605-290c469a71a5/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
golang.org/x/net v0.0.0-20220615171555-694bf12d69de/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.12.0 h1:cfawfvKITfUsFCeJIHJrbSxpeu/E81khclypR0GVT50=
golang.org/x/net v0.12.0/go.mod h1:zEVYFnQC7m/vmpQFELhcD1EWkZlX69l4oqgmer6hfKA=
golang.org/x/sync v0.1.0 h1:wsuoTGHzEhffawBOhz5CYhcrV4IdKZbEyZjBMuTp12o=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190804053845-51ab0e2deafa/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220412211240-33da011f77ad/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220615213510-4f61da869c0c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.10.0 h1:SqMFp9UcQJZa+pmYuAKjd9xq1f0j5rLcDIk0mj4qAsA=
golang.org/x/sys v0.10.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210615171337-6886f2dfbf5b/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.6.0/go.mod h1:m6U89DPEgQRMq3DNkDClhWw02AUbt2daBVO4cn4Hv9U=
golang.org/x/term v0.10.0 h1:3R7pNqamzBraeqj/Tj8qt1aQ2HpmlC+Cx/qL/7hn4/c=
golang.org/x/term v0.10.0/go.mod h1:lpqdcUyK/oCiQxvxVrppt5ggO2KCZ5QblwqPnfZ6d5o=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/text v0.11.0 h1:LAntKIrcmeSKERyiOh0XMV39LXS8IE9UL2yP7+f5ij4=
golang.org/x/text v0.11.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.6.0 h1:BOw41kyTf3PuCW1pVQf8+Cyg8pMlkYB1oo9iJ6D/lKM=
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

```

## /img/1.png

Binary file available at https://raw.githubusercontent.com/musana/CF-Hero/refs/heads/main/img/1.png

## /img/2.png

Binary file available at https://raw.githubusercontent.com/musana/CF-Hero/refs/heads/main/img/2.png

## /img/a

``` path="/img/a" 


```

## /img/cf-heroo.jpg

Binary file available at https://raw.githubusercontent.com/musana/CF-Hero/refs/heads/main/img/cf-heroo.jpg

## /img/gopher-cf-hero.png

Binary file available at https://raw.githubusercontent.com/musana/CF-Hero/refs/heads/main/img/gopher-cf-hero.png

## /internal/config/config.go

```go path="/internal/config/config.go" 
package config

import (
	"fmt"
	"os"

	"github.com/musana/cf-hero/pkg/models"
	"github.com/projectdiscovery/goflags"
	"gopkg.in/yaml.v2"
)

func ParseOptions() *models.Options {
	options := &models.Options{}
	flagSet := goflags.NewFlagSet()
	flagSet.SetDescription(`Unmask the origin IPs of Cloudflare-protected domains`)

	createGroup(flagSet, "General Options", "GENERAL OPTIONS",
		flagSet.IntVar(&options.Worker, "w", 16, "Worker count"),
		flagSet.StringVar(&options.File, "f", "", "Input file containing list of host/domain"),
	)

	createGroup(flagSet, "print options", "PRINT OPTIONS",
		flagSet.BoolVar(&options.CF, "cf", false, "Print domains behind Cloudflare"),
		flagSet.BoolVar(&options.NCF, "non-cf", false, "Print domains not behind Cloudflare"),
	)

	createGroup(flagSet, "sources", "SOURCES",
		flagSet.BoolVar(&options.Censys, "censys", false, "Include Censys in scanning"),
		flagSet.BoolVar(&options.SecurityTrails, "securitytrails", false, "Include SecurityTrails historical DNS records in scanning"),
		flagSet.BoolVar(&options.Shodan, "shodan", false, "Include Shodan historical DNS records in scanning"),
		flagSet.StringVar(&options.DomainList, "dl", "", "Domain list for sub/domain scanning"),
		flagSet.StringVar(&options.TargetDomain, "td", "", "Target domain for sub/domain scanning"),
	)

	createGroup(flagSet, "configuration", "CONFIGURATION",
		flagSet.StringVar(&options.HTTPMethod, "hm", "GET", "HTTP method."),
		flagSet.StringVar(&options.JA3, "ja3", "772,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,18-10-16-23-45-35-5-11-13-65281-0-51-43-17513-27,29-23-24,0", "JA3 String"),
		flagSet.StringVar(&options.UserAgent, "ua", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/113.0", "HTTP User-Agent"),
		flagSet.StringVar(&options.Proxy, "px", "", "HTTP proxy URL"),
	)

	_ = flagSet.Parse()

	return options
}

func createGroup(flagSet *goflags.FlagSet, groupName, description string, flags ...*goflags.FlagData) {
	flagSet.SetGroup(groupName, description)
	for _, currentFlag := range flags {
		currentFlag.Group(groupName)
	}
}

func ReadAPIKeys(source string) []string {
	home := os.Getenv("HOME")
	if home == "" {
		home = os.Getenv("USERPROFILE") // Windows için
	}

	configPath := home + "/.config/cf-hero.yaml"
	f, err := os.ReadFile(configPath)
	if err != nil {
		fmt.Printf("[!] Error reading config file %s: %v\n", configPath, err)
		return nil
	}

	var apiKeys map[string][]string
	err = yaml.Unmarshal(f, &apiKeys)
	if err != nil {
		fmt.Printf("[!] Error parsing YAML from %s: %v\n", configPath, err)
		return nil
	}

	keys, ok := apiKeys[source]
	if !ok {
		fmt.Printf("[!] No API keys found for source '%s' in %s\n", source, configPath)
		return nil
	}

	return keys
}

```

## /internal/dns/dns.go

```go path="/internal/dns/dns.go" 
package dns

import (
	"net"
	"regexp"

	"github.com/miekg/dns"
	"github.com/projectdiscovery/retryabledns"
)

func GetARecords(domain string) ([]net.IP, []net.IP) {
	var cfIPs []net.IP
	var nonCFIPs []net.IP

	ips, _ := net.LookupIP(domain)
	if len(ips) > 0 {
		for _, ip := range ips {
			if ip.To4() != nil {
				result, _ := IsInCloudflareIPRange(ip)
				if result {
					cfIPs = append(cfIPs, ip)
				} else {
					nonCFIPs = append(nonCFIPs, ip)
				}
			}
		}
	}
	return cfIPs, nonCFIPs
}

func GetTXTRecords(domain string) ([]string, error) {
	resolvers := []string{"1.1.1.1:53", "8.8.8.8:53", "8.8.4.4:53", "1.0.0.1:53"}
	retries := 3
	dnsClient, err := retryabledns.New(resolvers, retries)
	if err != nil {
		return nil, err
	}

	TXTRecords, err := dnsClient.Query(domain, dns.TypeTXT)
	if err != nil {
		return nil, err
	}

	return TXTRecords.TXT, nil
}

func ExtractIPAddresses(input string) []string {
	ipPattern := `\b(?:\d{1,3}\.){3}\d{1,3}\b`
	re := regexp.MustCompile(ipPattern)
	return re.FindAllString(input, -1)
}

func IsInCloudflareIPRange(aIP net.IP) (bool, net.IP) {
	cloudflareRanges := []string{
		"173.245.48.0/20",
		"103.21.244.0/22",
		"103.22.200.0/22",
		"103.31.4.0/22",
		"141.101.64.0/18",
		"108.162.192.0/18",
		"190.93.240.0/20",
		"188.114.96.0/20",
		"197.234.240.0/22",
		"198.41.128.0/17",
		"162.158.0.0/15",
		"104.16.0.0/13",
		"104.24.0.0/14",
		"172.64.0.0/13",
		"131.0.72.0/22",
	}

	for _, rangeStr := range cloudflareRanges {
		_, cidr, _ := net.ParseCIDR(rangeStr)
		if cidr.Contains(aIP) {
			return true, aIP
		}
	}

	return false, aIP
}

```

## /internal/http/client.go

```go path="/internal/http/client.go" 
package http

import (
	"crypto/tls"
	"fmt"
	"net"
	"net/http"
	"net/url"
	"strings"
	"time"

	"github.com/Danny-Dasilva/CycleTLS/cycletls"
	"github.com/hashicorp/go-retryablehttp"
	"golang.org/x/net/html"
)

func NewHTTPClient(proxy string, urlx string) *http.Client {
	tr := &http.Transport{
		MaxIdleConns:        20,
		MaxConnsPerHost:     20,
		MaxIdleConnsPerHost: 20,
		IdleConnTimeout:     time.Second * 2,
		DisableKeepAlives:   true,
		TLSClientConfig:     &tls.Config{InsecureSkipVerify: true},
		DialContext: (&net.Dialer{
			Timeout:   time.Second * 2,
			KeepAlive: time.Second * 2,
		}).DialContext,
	}

	if proxy != "" {
		if p, err := url.Parse(proxy); err == nil {
			tr.Proxy = http.ProxyURL(p)
		}
	}

	retryClient := retryablehttp.NewClient()
	retryClient.RetryMax = 2
	retryClient.RetryWaitMax = time.Second * 1
	retryClient.Logger = nil
	retryClient.HTTPClient.Transport = tr
	httpClient := retryClient.StandardClient()

	return httpClient
}

func RequestBuilderWithHost(url, hostHeader, httpMethod, userAgent string) *http.Request {
	req, _ := http.NewRequest(httpMethod, url, nil)
	req.Header.Add("User-Agent", userAgent)
	req.Header.Add("Connection", "Close")
	req.Header.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8")
	req.Host = hostHeader

	return req
}

func RequestBuilder(url, token, httpMethod, userAgent string) *http.Request {
	req, _ := http.NewRequest(httpMethod, url, nil)
	req.Header.Add("User-Agent", userAgent)
	req.Header.Add("Connection", "Close")
	req.Header.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8")
	req.Header.Add("Authorization", "Basic "+token)

	return req
}

func CycleTLSforJA3(url, ja3, userAgent, proxy string) (cycletls.Response, error) {
	client := cycletls.Init()

	response, err := client.Do(url, cycletls.Options{
		Body:            "",
		Ja3:             ja3,
		UserAgent:       userAgent,
		Timeout:         5,
		Proxy:           proxy,
		DisableRedirect: false,
		Headers: map[string]string{
			"Content-Type": "application/json",
		},
	}, "GET")

	return response, err
}

func GetHTMLTitle(doc *html.Node) string {
	var title string
	var traverse func(*html.Node)
	traverse = func(n *html.Node) {
		if n.Type == html.ElementNode && n.Data == "title" && n.FirstChild != nil {
			title = n.FirstChild.Data
			return
		}
		for c := n.FirstChild; c != nil; c = c.NextSibling {
			traverse(c)
		}
	}
	traverse(doc)
	return title
}

// CheckPort checks if a port is open on a host
func CheckPort(host string, port string) bool {
	timeout := time.Second * 2
	conn, err := net.DialTimeout("tcp", net.JoinHostPort(host, port), timeout)
	if err != nil {
		return false
	}
	if conn != nil {
		conn.Close()
		return true
	}
	return false
}

// GetHTMLTitleWithPortCheck tries to get HTML title with port checking
func GetHTMLTitleWithPortCheck(ip string, ja3, userAgent, proxy string) (string, error) {
	// First try HTTP (port 80)
	if CheckPort(ip, "80") {
		resp, err := CycleTLSforJA3("http://"+ip, ja3, userAgent, proxy)
		if err == nil && resp.Body != "" {
			reader := strings.NewReader(resp.Body)
			doc, err := html.Parse(reader)
			if err == nil {
				title := GetHTMLTitle(doc)
				if title != "" {
					return title, nil
				}
			}
		}
	}

	// If HTTP fails or returns empty title, try HTTPS (port 443)
	if CheckPort(ip, "443") {
		resp, err := CycleTLSforJA3("https://"+ip, ja3, userAgent, proxy)
		if err == nil && resp.Body != "" {
			reader := strings.NewReader(resp.Body)
			doc, err := html.Parse(reader)
			if err == nil {
				return GetHTMLTitle(doc), nil
			}
		}
	}

	return "", fmt.Errorf("no accessible ports found or no title available")
}

```

## /internal/scanner/scanner.go

```go path="/internal/scanner/scanner.go" 
package scanner

import (
	"encoding/base64"
	"encoding/json"
	"fmt"
	"io"
	"net"
	"net/http"
	neturl "net/url"
	"strings"
	"sync"
	"time"

	"github.com/gammazero/workerpool"
	"github.com/musana/cf-hero/internal/config"
	"github.com/musana/cf-hero/internal/dns"
	httpClient "github.com/musana/cf-hero/internal/http"
	"github.com/musana/cf-hero/pkg/models"
	"github.com/schollz/progressbar/v3"
)

type Scanner struct {
	Options *models.Options
	URLs    []string
	Domains []string
	Bar     *progressbar.ProgressBar
	mu      sync.Mutex
	Stats   struct {
		Total     int
		Behind    int
		NotBehind int
	}
}

func New(options *models.Options, urls []string, domains []string) *Scanner {
	return &Scanner{
		Options: options,
		URLs:    urls,
		Domains: domains,
	}
}

func (s *Scanner) PreScan() {
	fmt.Println("\n[*] Pre-scanning domains to identify Cloudflare protected ones...")
	total := len(s.URLs)
	processed := 0

	fmt.Print("\r[*] Pre-Scanning: 0/" + fmt.Sprint(total))

	wp := workerpool.New(s.Options.Worker)
	var wg sync.WaitGroup

	for _, url := range s.URLs {
		url := url // capture variable
		wg.Add(1)
		wp.Submit(func() {
			defer wg.Done()
			domain := strings.Split(url, "//")[1]
			cfIPs, _ := dns.GetARecords(domain)

			s.mu.Lock()
			processed++
			s.Stats.Total++
			if len(cfIPs) > 0 {
				s.Stats.Behind++
			} else {
				s.Stats.NotBehind++
			}
			fmt.Print("\r[*] Progress: " + fmt.Sprint(processed) + "/" + fmt.Sprint(total))
			s.mu.Unlock()
		})
	}

	wg.Wait()
	wp.StopWait()

	fmt.Printf("\n[+] Found %d/%d domains behind Cloudflare\n\n", s.Stats.Behind, s.Stats.Total)

	if s.Stats.Behind > 0 {
		s.Bar = progressbar.NewOptions(s.Stats.Behind,
			progressbar.OptionEnableColorCodes(true),
			progressbar.OptionShowCount(),
			progressbar.OptionSetWidth(40),
			progressbar.OptionSetDescription("[cyan][*][reset] Scanning Cloudflare protected domains..."),
			progressbar.OptionSetTheme(progressbar.Theme{
				Saucer:        "[green]=[reset]",
				SaucerHead:    "[green]>[reset]",
				SaucerPadding: " ",
				BarStart:      "[",
				BarEnd:        "]",
			}))
	}
}

func (s *Scanner) Start(url string) {
	if s.Options.CF || s.Options.NCF {
		s.printDomains(url)
		return
	}

	domain := strings.Split(url, "//")[1]
	cfIPs, nonCFIPs := dns.GetARecords(domain)

	if len(cfIPs) > 0 {
		actualHTMLTitle, _ := s.getHTMLTitle(url)

		if len(nonCFIPs) > 0 {
			s.checkARecords(url, nonCFIPs, cfIPs[0], actualHTMLTitle)
			fmt.Println("Found:", "URL", url, "CF IP", cfIPs, "NON-CF-IP:", nonCFIPs)
		}

		s.getTXTRecords(domain, url, cfIPs[0], actualHTMLTitle)

		if s.Options.Censys {
			s.censysSearch(domain, url, cfIPs[0], actualHTMLTitle)
		}

		if s.Options.SecurityTrails {
			s.securityTrailsSearch(domain, url, cfIPs[0], actualHTMLTitle)
		}

		if s.Options.Shodan {
			s.shodanSearch(domain, url, cfIPs[0], actualHTMLTitle)
		}

		if s.Options.DomainList != "" && s.Options.TargetDomain != "" {
			s.checkDomainList(url, cfIPs[1], actualHTMLTitle)
		}

		if s.Bar != nil {
			s.Bar.Add(1)
		}
	}
}

func (s *Scanner) printDomains(url string) {
	domain := strings.Split(url, "//")[1]
	cfIPs, nonCFIPs := dns.GetARecords(domain)

	if s.Options.CF {
		if len(cfIPs) > 0 {
			fmt.Println(url)
		}
	}
	if s.Options.NCF {
		if len(nonCFIPs) > 0 {
			fmt.Println(url)
		}
	}
}

func (s *Scanner) checkDomainList(url string, cfIP net.IP, actualHTMLTitle string) {
	for _, d := range s.Domains {
		parseIt := strings.Split(d, "//")
		domain := parseIt[1]

		targetDomain := strings.Split(s.Options.TargetDomain, "//")[1]

		_, nonCFIPs := dns.GetARecords(domain)

		if len(nonCFIPs) > 0 {
			for _, ip := range nonCFIPs {
				htmlTitle := s.checkHTMLTitle("http://"+ip.String(), targetDomain)
				if htmlTitle == actualHTMLTitle {
					s.printResult(url, cfIP, ip, "DNS A Record", actualHTMLTitle)
				}
			}
		}
	}
}

func (s *Scanner) checkHTMLTitle(urlStr string, hostHeader string) string {
	// URL'den host kısmını çıkar
	parsedURL, err := neturl.Parse(urlStr)
	if err != nil {
		return ""
	}

	title, err := httpClient.GetHTMLTitleWithPortCheck(parsedURL.Host, s.Options.JA3, s.Options.UserAgent, s.Options.Proxy)
	if err != nil {
		return ""
	}
	return title
}

func (s *Scanner) getHTMLTitle(urlStr string) (string, error) {
	// URL'den host kısmını çıkar
	parsedURL, err := neturl.Parse(urlStr)
	if err != nil {
		return "", err
	}

	return httpClient.GetHTMLTitleWithPortCheck(parsedURL.Host, s.Options.JA3, s.Options.UserAgent, s.Options.Proxy)
}

func (s *Scanner) checkARecords(url string, ips []net.IP, cfIP net.IP, actualHTMLTitle string) {
	for _, ip := range ips {
		s.compareTitle(url, ip, cfIP, "A - Record", actualHTMLTitle)
	}
}

func (s *Scanner) getTXTRecords(domain, url string, cfIP net.IP, actualHTMLTitle string) {
	txtRecords, err := dns.GetTXTRecords(domain)
	if err != nil {
		return
	}

	var extractedIPs []string
	for _, txt := range txtRecords {
		extractedIP := dns.ExtractIPAddresses(txt)
		if len(extractedIP) > 0 {
			for _, ipAddress := range extractedIP {
				if !strings.Contains(strings.Join(extractedIPs, ","), ipAddress) {
					extractedIPs = append(extractedIPs, ipAddress)
				}
			}

			for _, ipx := range extractedIPs {
				netIP := net.ParseIP(ipx)
				if netIP.To4() != nil {
					s.compareTitle(url, netIP, cfIP, "TXT - DNS Record", actualHTMLTitle)
				}
			}
		}
	}
}

func (s *Scanner) censysSearch(domain, url string, cfIP net.IP, actualHTMLTitle string) {
	key := config.ReadAPIKeys("censys")[0]
	keyToBytes := []byte(key)
	token := base64.StdEncoding.EncodeToString(keyToBytes)

	censysURL := "https://search.censys.io/api/v2/hosts/search?q=" + domain + "&per_page=50&virtual_hosts=EXCLUDE"
	client := httpClient.NewHTTPClient(s.Options.Proxy, url)
	resp, err := client.Do(httpClient.RequestBuilder(censysURL, token, s.Options.HTTPMethod, s.Options.UserAgent))
	if err != nil {
		return
	}
	defer resp.Body.Close()

	var data models.CensysJSON
	if err := json.NewDecoder(resp.Body).Decode(&data); err != nil {
		return
	}

	for _, cip := range data.Result.Hits {
		censysIP := net.ParseIP(cip.IP)
		if censysIP.To4() != nil {
			result, _ := dns.IsInCloudflareIPRange(censysIP)
			if !result {
				s.compareTitle(url, censysIP, cfIP, "Censys", actualHTMLTitle)
			}
		}
	}
}

func (s *Scanner) securityTrailsSearch(domain, url string, cfIP net.IP, actualHTMLTitle string) {
	keys := config.ReadAPIKeys("securitytrails")
	if len(keys) == 0 {
		fmt.Println("[!] SecurityTrails API key not found in ~/.config/cf-hero.yaml")
		return
	}
	key := keys[0]
	if key == "" {
		fmt.Println("[!] SecurityTrails API key is empty in ~/.config/cf-hero.yaml")
		return
	}

	apiURL := fmt.Sprintf("https://api.securitytrails.com/v1/history/%s/dns/a", domain)
	client := httpClient.NewHTTPClient(s.Options.Proxy, url)

	req := httpClient.RequestBuilder(apiURL, "", s.Options.HTTPMethod, s.Options.UserAgent)
	req.Header.Set("APIKEY", key)
	req.Header.Set("Accept", "application/json")

	resp, err := client.Do(req)
	if err != nil {
		fmt.Printf("[!] Error making request to SecurityTrails API: %v\n", err)
		return
	}
	defer resp.Body.Close()

	var data models.SecurityTrailsResponse
	if err := json.NewDecoder(resp.Body).Decode(&data); err != nil {
		fmt.Printf("[!] Error decoding SecurityTrails response: %v\n", err)
		return
	}

	fmt.Printf("\n[*] SecurityTrails historical DNS records for %s:\n", domain)
	for _, record := range data.Records {
		for _, value := range record.Values {
			//fmt.Printf("[+] IP: %s (First seen: %s, Last seen: %s)\n",
			//	value.IP, record.FirstSeen, record.LastSeen)

			ip := net.ParseIP(value.IP)
			if ip != nil && ip.To4() != nil {
				result, _ := dns.IsInCloudflareIPRange(ip)
				if !result {
					s.compareTitle(url, ip, cfIP, "SecurityTrails", actualHTMLTitle)
				}
			}
		}
	}
	fmt.Println()
}

func (s *Scanner) shodanSearch(domain, url string, cfIP net.IP, actualHTMLTitle string) {
	keys := config.ReadAPIKeys("shodan")
	if len(keys) == 0 {
		fmt.Println("[!] Shodan API key not found in ~/.config/cf-hero.yaml")
		return
	}
	key := keys[0]
	if key == "" {
		fmt.Println("[!] Shodan API key is empty in ~/.config/cf-hero.yaml")
		return
	}

	maxRetries := 5
	retryCount := 0
	var resp *http.Response
	var err error

	apiURL := fmt.Sprintf("https://api.shodan.io/dns/domain/%s?key=%s&history=true", domain, key)
	client := httpClient.NewHTTPClient(s.Options.Proxy, url)
	req := httpClient.RequestBuilder(apiURL, "", s.Options.HTTPMethod, s.Options.UserAgent)
	req.Header.Set("Accept", "application/json")

	// Retry loop
	for retryCount < maxRetries {
		resp, err = client.Do(req)
		if err == nil && resp.StatusCode == 200 {
			break
		}

		if resp != nil {
			resp.Body.Close()
		}

		retryCount++
		if retryCount == maxRetries {
			fmt.Printf("[!] Error making request to Shodan API after %d retries: %v\n", maxRetries, err)
			return
		}

		// Exponential backoff: 1s, 2s, 4s, 8s, 16s
		waitTime := time.Duration(1<<uint(retryCount-1)) * time.Second
		fmt.Printf("[*] Retrying Shodan API request in %v (attempt %d/%d)...\n", waitTime, retryCount, maxRetries)
		time.Sleep(waitTime)
	}
	defer resp.Body.Close()

	// Check response status
	if resp.StatusCode != 200 {
		bodyBytes, _ := io.ReadAll(resp.Body)
		fmt.Printf("[!] Shodan API returned non-200 status code %d: %s\n", resp.StatusCode, string(bodyBytes))
		return
	}

	var data models.ShodanDNSHistoryResponse
	if err := json.NewDecoder(resp.Body).Decode(&data); err != nil {
		fmt.Printf("[!] Error decoding Shodan response: %v\n", err)
		return
	}

	fmt.Printf("\n[*] Shodan historical DNS records for %s:\n", domain)
	for _, record := range data.Data {
		if record.Type == "A" {
			fmt.Printf("[+] IP: %s (First seen: %s, Last seen: %s)\n",
				record.Value, record.FirstSeen, record.LastSeen)

			ip := net.ParseIP(record.Value)
			if ip != nil && ip.To4() != nil {
				result, _ := dns.IsInCloudflareIPRange(ip)
				if !result {
					s.compareTitle(url, ip, cfIP, "Shodan", actualHTMLTitle)
				}
			}
		}
	}
	fmt.Println()
}

func (s *Scanner) compareTitle(url string, ip net.IP, cfIP net.IP, source string, actualHTMLTitle string) {
	foundIPTitle, _ := httpClient.GetHTMLTitleWithPortCheck(ip.String(), s.Options.JA3, s.Options.UserAgent, s.Options.Proxy)

	if actualHTMLTitle == foundIPTitle {
		s.printResult(url, cfIP, ip, source, actualHTMLTitle)
	}
}

func (s *Scanner) printResult(url string, cfIP net.IP, realIP interface{}, source, htmlTitle string) {
	// ANSI color codes
	colors := struct {
		green   string
		cyan    string
		yellow  string
		red     string
		blue    string
		magenta string
		reset   string
		bold    string
	}{
		green:   "\033[32m",
		cyan:    "\033[36m",
		yellow:  "\033[33m",
		red:     "\033[31m",
		blue:    "\033[34m",
		magenta: "\033[35m",
		reset:   "\033[0m",
		bold:    "\033[1m",
	}

	banner := `
╭─────────────────────────────────────────────────────────────
│ %s%s[!] Real IP Found%s
├─────────────────────────────────────────────────────────────
│ %s%s Target Domain%s    │ %s%s%s			
│ %s%s CloudFlare IP%s    │ %s%s%s
│ %s%s Real IP%s          │ %s%v%s
│ %s%s Source%s           │ %s%s%s
│ %s%s HTML Title%s       │ %s%s%s
╰─────────────────────────────────────────────────────────────
`
	fmt.Printf(banner,
		colors.bold, colors.green, colors.reset,
		colors.bold, colors.blue, colors.reset, colors.yellow, url, colors.reset,
		colors.bold, colors.blue, colors.reset, colors.yellow, cfIP, colors.reset,
		colors.bold, colors.blue, colors.reset, colors.yellow, realIP, colors.reset,
		colors.bold, colors.blue, colors.reset, colors.magenta, source, colors.reset,
		colors.bold, colors.blue, colors.reset, colors.cyan, htmlTitle, colors.reset,
	)
}

```

## /internal/utils/utils.go

```go path="/internal/utils/utils.go" 
package utils

import (
	"bufio"
	"os"
)

func ReadFromStdin() []string {
	var urls []string
	scanner := bufio.NewScanner(os.Stdin)
	scanner.Split(bufio.ScanLines)

	for scanner.Scan() {
		urls = append(urls, scanner.Text())
	}
	return urls
}

func ReadFromFile(fileName string) []string {
	var fileContent []string
	file, err := os.Open(fileName)
	if err != nil {
		return nil
	}
	defer file.Close()

	scanner := bufio.NewScanner(file)
	for scanner.Scan() {
		fileContent = append(fileContent, scanner.Text())
	}
	return fileContent
}

func Contains(slice []string, elements string) bool {
	for _, s := range slice {
		if elements == s {
			return true
		}
	}
	return false
}

func Banner() string {
	return `
        ____         __                  
  _____/ __/        / /_  ___  _________ 
 / ___/ /__  ___   / __ \/ _ \/ ___/ __ \
/ /__/ ___/ (___) / / / /  __/ /  / /_/ /
\___/_/          /_/ /_/\___/_/   \____/ 

                                @musana
_____________________________________________

`
}

```

## /pkg/models/models.go

```go path="/pkg/models/models.go" 
package models

import "time"

type Options struct {
	File           string
	Worker         int
	Version        bool
	HTTPMethod     string
	UserAgent      string
	Proxy          string
	DomainList     string
	TargetDomain   string
	JA3            string
	CF             bool
	NCF            bool
	Censys         bool
	SecurityTrails bool
	Shodan         bool
}

type CensysJSON struct {
	Code   int    `json:"code"`
	Status string `json:"status"`
	Result struct {
		Hits []struct {
			IP            string    `json:"ip"`
			LastUpdatedAt time.Time `json:"last_updated_at"`
		} `json:"hits"`
		Links struct {
			Next string `json:"next"`
			Prev string `json:"prev"`
		} `json:"links"`
	} `json:"result"`
}

type SecurityTrailsResponse struct {
	Records []struct {
		Values []struct {
			IP string `json:"ip"`
		} `json:"values"`
		Type      string `json:"type"`
		FirstSeen string `json:"first_seen"`
		LastSeen  string `json:"last_seen"`
	} `json:"records"`
	Meta struct {
		TotalRecords int `json:"total_records"`
	} `json:"meta"`
}

type ShodanDNSHistoryResponse struct {
	Data []struct {
		Type      string `json:"type"`
		Value     string `json:"value"`
		LastSeen  string `json:"last_seen"`
		FirstSeen string `json:"first_seen"`
	} `json:"data"`
}

```


The better and more specific the context, the better the LLM can follow instructions. If the context seems verbose, the user can refine the filter using uithub. Thank you for using https://uithub.com - Perfect LLM context for any GitHub repo.