Recon Commands

Với:

• urls.txt: danh sách các URLs (có thể có dạng domain).
• tlds.txt: danh sách các top-level domains (wildcards).

Endpoint Scraping

cat tlds.txt | gau --fc 404 --fp --o /tmp/urls.gau.txt --threads 20 --verbose --providers wayback,commoncrawl,otx,urlscan
 
cat /tmp/urls.gau.txt | anew urls.gau.txt

cat tlds.txt | xargs -t -I{} urx --config ~/.config/urx/config.toml --normalize-url -o /tmp/{}.urls.urx.txt {}
 
cat /tmp/*.urls.urx.txt | anew urls.urx.txt

waymore -i tlds.txt -mode U -oU /tmp/urls.waymore.txt
 
cat /tmp/urls.waymore.txt | anew urls.waymore.txt

urlfinder -d tlds.txt -us tlds.txt -pc ~/.config/urlfinder/provider-config.yaml -o /tmp/urls.urlfinder.txt

cat /tmp/urls.urlfinder.txt | anew urls.urlfinder.txt

cat tlds.txt | waybackurls | tee /tmp/urls.waybackurls.txt
 
cat /tmp/urls.waybackurls.txt | anew urls.waybackurls.txt

Optional (mostly run on targets that have sharing feature)

python ../misc/generate-urlhunter-keywords.py -dL tlds.txt -o urlhunter-queries.txt
urlhunter -k urlhunter-queries.txt -d 2025 -o urls.urlhunter.txt --rm
cat tlds.txt | xargs -I{} grep "\\.{}\/" urls.urlhunter.txt > /tmp/urls.urlhunter.tmp
 
cat /tmp/urls.urlhunter.tmp | anew urls.urlhunter.txt

Tip

Filter based on the scope:

cat urls.*.txt > /tmp/urls.txt
sed -i '' 's/^[^h]/https:\/\/&/' /tmp/urls.txt # Add https to domains
cat tlds.txt | xargs -I{} grep '{}' /tmp/urls.txt > /tmp/urls.tmp
cat /tmp/urls.tmp | anew urls.txt

Tip

Use urless and uro to reduce URLs.

uro -i /tmp/urls.txt -o /tmp/urls.txt
urless -i /tmp/urls.txt -o /tmp/urls.txt
cat /tmp/urls.txt | anew urls.txt

Tip

Use unfurl to extract domains:

cat urls.txt | unfurl domains | anew subdomains.urls.txt

Important

Need to inspect the found URLs manually.

Subdomain Enumeration

Flow: passive - OSINT - permutation - active - resolve

Passive

chaos-client -dL tlds.txt -o /tmp/chaos.txt # Has both wildcards and subdomains
 
cat /tmp/chaos.txt | grep '\*' | sed 's/\*\.//g' | anew wildcards.txt # Extract wildcards
cat /tmp/chaos.txt | grep -vEi '\*|^\.' | anew subdomains.chaos.txt # Extract subdomains

subfinder -dL tlds.txt -o /tmp/subfinder.json
 
cat /tmp/subfinder.json | jq -r '.host' | anew subdomains.subfinder.txt # Extract subdomains
cat /tmp/subfinder.json | jq -r '.ip' | anew ips.subfinder.txt # Extract ips

findomain --config ~/.config/findomain/config.yml --resolvers ~/repos/bug-bounty/misc/resolvers.txt --double-dns-check -f tlds.txt -u /tmp/subdomains.findomain.txt
cat /tmp/subdomains.findomain.txt | anew subdomains.findomain.txt

samoscout -dL tlds.txt -c ~/.config/samoscout/config.yaml -json -o /tmp/samoscout.json # Remeber to delete the last line
 
cat /tmp/samoscout.json | jq -r '.host' | grep '\*' | sed 's/\*\.//' | anew wildcards.txt # Extract wildcards
cat /tmp/samoscout.json | jq -r '.host' | grep -v '\*\.' | anew subdomains.samoscout.txt # Extract subdomains

export VT_API_KEY=d306e0e80e34c48f073c100e9caa486e1b5448335fe735efcf21cfcff886c39c
cat tlds.txt | assetfinder | tee /tmp/subdomains.assetfinder.txt # Need to filter manually
 
cat tlds.txt | xargs -I{} grep '\.{}' /tmp/subdomains.assetfinder.txt | anew subdomains.assetfinder.txt
cat subdomains.assetfinder.txt | grep '\*\.' | sed 's/\*\.//' | anew wildcards.txt # Extract wildcards.txt
cat subdomains.assetfinder.txt | grep -v '\*\.' | sponge subdomains.assetfinder.txt # Remove wildcards.txt in subdomains file

Only on linux:

bbot -t tlds.txt -w tlds.txt -p cloud-enum -o ./bbot --exclude-cdn --json -em github_codesearch -m git,gitlab_com,dockerhub,google_playstore,jadx,apkpure,git_clone,skymem,emailformat,pgp,bypass403,ffuf,fingerprintx,host_header,hunt,reflected_parameters,paramminer_getparams,smuggler --allow-deadly -om csv,json,subdomains,emails,web_parameters
 
cp ./bbot/enum/subdomains.txt ./subdomains.bbot.txt
cp ./bbot/enum/emails.txt ./emails.bbot.txt
# Manually inspect other results such as open ports, asn, ...

Optional:

cat tlds.txt | xargs -t -I{} github-subdomains -d {} -t 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 -e -o {}.subdomains.github.txt -raw -q # Need to filter manually
 
cat *.subdomains.github.txt | anew subdomains.github.txt

Permutation

cat subdomains.*.txt > /tmp/subdomains.txt
 
cat tlds.txt | xargs -t -I{} regulator -d {} -f /tmp/subdomains.txt -o /tmp/{}.regulator.txt -r /tmp/{}.regulator.rules.txt
 
cat /tmp/*.regulator.txt | anew subdomains.regulator.txt
split -n 5 -d -a 1 subdomains.regulator.txt subdomains.regulator. # Split for uploading to github
cat /tmp/*.regulator.rules.txt | anew regulator.rules.txt

Resolving

cat subdomains.*.txt > /tmp/subdomains.txt
 
puredns resolve /tmp/subdomains.txt --rate-limit 10000 --rate-limit-trusted 100 --debug --write-massdns /tmp/massdns.txt --write-wildcards /tmp/wildcards.txt  -w /tmp/subdomains.resolved.txt
 
cat /tmp/massdns.txt | anew massdns.txt
cat /tmp/wildcards.txt | anew wildcards.txt
cat /tmp/subdomains.resolved.txt | anew subdomains.resolved.txt

Active

cat wildcards.txt tlds.txt > /tmp/wildcards.txt
 
puredns bruteforce ~/repos/bug-bounty/misc/2m-subdomains.txt -d tlds.txt -w /tmp/subdomains.puredns.txt --rate-limit 10000 --rate-limit-trusted 100 --debug --write-massdns /tmp/massdns.txt --bin $(which massdns)
 
cat /tmp/massdns.txt | anew massdns.txt
cat /tmp/subdomains.puredns.txt | anew subdomains.puredns.txt

Tip

Tìm subdomain takeover bằng cách kiểm tra xem các CNAME records có khớp với list CNAME từ can-i-take-over-xyz hay không:

cat ../misc/can-i-take-over-xyz.cnames.txt | xargs -I{} grep '{}' massdns.txt | awk '{print $1}' | sed 's/\.$//' | anew subdomain-takeover.txt # Get the vulnerable domains from massdns output

Verify subdomain takeover bằng dig:

cat subdomain-takeover.txt | xargs -I{} dig {} | grep NXDOMAIN

Probing & Screenshot

Probing subdomains:

cat subdomains.resolved.txt subdomains.puredns.txt > /tmp/subdomains.alive.txt
cat tlds.txt | xargs -I{} grep '{}' /tmp/subdomains.alive.txt | sort -u | sponge /tmp/subdomains.alive.tmp # Filter out of scope domains
 
httpx -l /tmp/subdomains.alive.tmp -resolvers ../misc/resolvers.txt -o httpx.subdomains.json -rl 500 -aname <target_name> # Target name can be mercedes, capital, etc. Subsequent runs will use -aid instead of -aname.
 
cat httpx.subdomains.json | jq -r '.url' | anew sites.txt # Extract alive sites
xargs -a tlds.txt -I{} grep '{}$' sites.txt | sponge sites.txt
cat httpx.subdomains.json | jq -r '.a[]' | anew ips.txt # Extract IPs

Screenshot:

cat sites.txt | goverview screen -j -o goverview-sites -L -c 20 --screen-timeout 60 -S screenshot-sites-summary.json

Finding interesting screenshots with eyeballer (optional):

eyeballer --weights ~/repos/eyeballer/bishop-fox-pretrained-v3.h5 predict ./goverview/screenshots

Find more subdomains with csprecon:

cat sites.txt | unfurl domains | anew /tmp/domains.txt
cat /tmp/urls.txt | unfurl domains | anew /tmp/domains.txt
csprecon -l /tmp/domains.txt -d tlds.txt -j -o /tmp/subdomains.csprecon.json
cat /tmp/subdomains.csprecon.json | jq -r '.URL' | anew subdomains.csprecon.txt

Output of csprecon will be used for the next round.

IP Discovery

Trích xuất từ output của massdns.

awk '$2 == "A" {print $3}' massdns.txt | ipfilter | anew ips.txt

Trích xuất từ output của subfinder.

cat /tmp/subfinder.json | jq -r '.ip' | ipfilter | anew ips.txt

Tip

Sử dụng nuclei với template http/technologies/tech-detect/yaml để detect technologies của các IPs để xem có IP nào không nằm sau WAF hay không.

Crawling

Ở trên alive sites:

cat tlds.txt | xargs -t -I{} gospider -S sites.txt -o gospider -L 0 -t 10 -d 3 --verbose --whitelist {} --include-subs --subs --json --no-redirect
 
cat gospider/* | grep -v 'code-200' | jq -r '.output' > /tmp/endpoints.gospider.txt
cat tlds.txt | xargs -t -I{} grep '{}' /tmp/endpoints.gospider.txt | grep -Ei '^http|^ws' | anew endpoints.gospider.txt
 
# Filter out mime types endpoints

cat sites.txt | hakrawler -d 3 -s -t 50 -json -u > /tmp/endpoints.hakrawler.json
cat /tmp/endpoints.hakrawler.json | jq -r '.URL' > /tmp/endpoints.hakrawler.txt
cat tlds.txt | xargs -t -I{} grep '{}' /tmp/endpoints.hakrawler.txt | grep -Ei '^http|^ws' | anew endpoints.hakrawler.txt

xnLinkFinder -i sites.txt -o /tmp/endpoints.xnlinkfinder.txt -owl words.xnlinkfinder.txt -d 5 --include -cl -v --scope-filter tlds.txt --memory-threshold 99 --processes 100 -s429 --no-banner
cat /tmp/endpoints.xnlinkfinder.txt | grep -Ei '^http|^ws' | sponge endpoints.xnlinkfinder.txt

Tip

Filter and declutter endpoints just like urls.

cat endpoints.*.txt > /tmp/endpoints.txt
uro -i /tmp/endpoints.txt -o /tmp/endpoints.txt
urless -i /tmp/endpoints.txt -o /tmp/endpoints.txt
cat /tmp/endpoints.txt | grep -Ei '^http|^ws' | anew endpoints.txt

Tip

Extract JavaScript

cat endpoints.txt | grep '\.js

subjs -i sites.txt | anew js.txt
 
# Feed to jsmon.sh

Important

Need to inspect the found endpoints manually.

Port Scanning

Important

Cần phải loại bỏ các private IPs trong input list (sử dụng aleister1102/ipfilter).

Filter các IP thuộc về cloud, WAF, CDN:

cat /tmp/ips | cdnstrip | cdncheck -exclude | sort -u | sponge /tmp/ips

Sử dụng shodan để tìm thông tin hostname của các IP nhằm loại bỏ các IP bên ngoài scope.

cat /tmp/ips | xargs -I{} shodan host {} -S

Scan port:

naabu -l /tmp/ips -aname <asset_name>-passive -passive -rate 500 # passive
naabu -l /tmp/ips -aname <asset_name>-active

Important

Cần chạy passsive scanning ít nhất 3 lần để đảm bảo không thiếu port từ Shodan DB. Bổ sung vào asset đã có trên ProjectDiscovery Cloud thông qua flag -aid có giá trị là ID của asset đã upload.

Vuln Scanning

nuclei -l sites.txt -scan-all-ips -skip-format-validation -no-httpx -nc -follow-redirects -iserver insomnia1102.online -concurrency 50 -rl 10000 -stats -http-stats -tp recommended -o -j nuclei.json -dashboard

Directory Bruteforce

feroxbuster -u https://salt-ece-nonprod.azure.mercedes-benz.com -w ~/Repos/bug-bounty/misc/lists/dirsearch.txt --smart --random-agent -Q data=data -C 404 --unique --redirects -k --auto-tune --collect-extensions --json -o /tmp/ferox.json
 
cat /tmp/urls | xargs -t -I{} ffuf -u '{}/FUZZ' -w ../misc/lists/dirsearch.txt -ac -recursion -recursion-depth 3 -o /tmp/ffuf.json -of json -fc 403,404
 
ffuf -u https://oneai.amap.npr.vida.aws.ve.dvb.corpinter.net/FUZZ -w ~/Tools/wordlists/trickest/inventory/levels/level1.txt -config ~/.config/ffuf/ffufrc
 
cat /tmp/urls | xargs -t -I{} dirsearch -u '{}' -t 40 -r -R 3 -x 403,404,503 --random-agent -O json -o /tmp/dirsearch.json

OSINT

Metabigor

The following commands need to be executed manually:

cat tlds.txt | metabigor cert --json -o /tmp/certs.json
xargs -a tlds.txt -I{} jq -r --arg tld {} 'select(.Domain | test("\\." + $tld + "$"))' /tmp/certs.json | sponge /tmp/certs.json # Filter out of scope domains

Trích xuất org name:

cat /tmp/certs.json | jq -r '.CertInfo | match("CN=([^,]+)") | .captures[0].string' | sort -u

Trích xuất subdomains:

cat /tmp/certs.json | jq -r '.Domain' | sort -u | anew subdomains.certs.txt

Sử dụng metabigor với Org Name để tìm IP:

metabigor net -i "Org Name" --org --json -o cidr.json
 
cat cidr.json | jq -r '.CIDR' | grep -v '::' | mapcidr | anew ips.txt # Map CIDR (exclude IPv6) to IPs

Bbot

Kiểm tra output của Bbot đã chạy ở bước Passive bao gồm emails, code repository, APK, buckets, …

Dorking

Sử dụng Bug Bounty Helper.

Sử dụng uncover:

python ../misc/generate-uncover-queries.py -dL tlds.txt -o ./uncover-queries
./uncover-queries/run-uncover.sh

Warning

Engines that need to use manually:

• Hunterhow

FOFA

NOTE: Use favicon hashes in search results to find related assets

Resources:

• Google dorking cheatsheet: Google dork cheatsheet · GitHub

• Brave search engine: Search Operators | Brave Search

• DuckDuckGo search engine: How to use advanced syntax on DuckDuckGo Search - DuckDuckGo Help Pages

• URL Scan:

  page.domain:(domain1.com OR domain2.com)
  page.url:domain.com
  page.title:"Org Name"
  page.asnname:"Org Name"

• Dùng để search code snippet: Search Engine for Source Code - PublicWWW.com. Ví dụ: PublicWWW - PublicWWW.com.

Linux

sed -E 's/\x1B\[[0-9;]*m//g' sites.subdomains.httpx.txt # remove UNICODE chars

# 1. Stash current working changes
cd /path/to/repo && git stash push --include-untracked -m "pre-bfg cleanup"
 
# 2. Remove large folders from git history
bfg --delete-folders "{folder1,folder2,folder3}" --no-blob-protection .git
 
# 3. Clean up reflog and garbage collection
cd /path/to/repo && git reflog expire --expire=now --all && git repack -Ad && git prune --expire=now
 
# 4. Force push the cleaned history
git push --force-with-lease origin master

sed -i '' 's/^[^h]/https:\/\/&/' urls.txt # Add https to domains in a file

find . -maxdepth 1 -type f -empty -delete # Delete empty files in the current folder

find . -type f ! -name "*.*" -exec sh -c 'file "$1" | grep -q "text" && [ "$(wc -l < "$1")" -eq 1 ] && rm "$1"' _ {} \; # Remove text files that have only one line

rg -l0 'PATTERN' . | xargs -0 rm -- # Remove files in the current folder that match a pattern

jq -s 'group_by(.ip) | map(.[0]) | .[]' shodan.json | sponge shodan.json # Dedup json object in jsonl file.

| grep -vEi ’*|lib’ | anew js.txt

subjs -i sites.txt | anew js.txt
 
# Feed to jsmon.sh

Need to inspect the found endpoints manually.

Port Scanning

Cần phải loại bỏ các private IPs trong input list (sử dụng `aleister1102/ipfilter`).

Filter các IP thuộc về cloud, WAF, CDN:

cat /tmp/ips | cdnstrip | cdncheck -exclude | sort -u | sponge /tmp/ips

Sử dụng shodan để tìm thông tin hostname của các IP nhằm loại bỏ các IP bên ngoài scope.

cat /tmp/ips | xargs -I{} shodan host {} -S

Scan port:

naabu -l /tmp/ips -aname <asset_name>-passive -passive -rate 500 # passive
naabu -l /tmp/ips -aname <asset_name>-active

Cần chạy passsive scanning ít nhất 3 lần để đảm bảo không thiếu port từ Shodan DB. Bổ sung vào asset đã có trên ProjectDiscovery Cloud thông qua flag `-aid` có giá trị là ID của asset đã upload.

Vuln Scanning

nuclei -l sites.txt -scan-all-ips -skip-format-validation -no-httpx -nc -follow-redirects -iserver insomnia1102.online -concurrency 50 -rl 10000 -stats -http-stats -tp recommended -o -j nuclei.json -dashboard

Directory Bruteforce

feroxbuster -u https://salt-ece-nonprod.azure.mercedes-benz.com -w ~/Repos/bug-bounty/misc/lists/dirsearch.txt --smart --random-agent -Q data=data -C 404 --unique --redirects -k --auto-tune --collect-extensions --json -o /tmp/ferox.json
 
cat /tmp/urls | xargs -t -I{} ffuf -u '{}/FUZZ' -w ../misc/lists/dirsearch.txt -ac -recursion -recursion-depth 3 -o /tmp/ffuf.json -of json -fc 403,404
 
ffuf -u https://oneai.amap.npr.vida.aws.ve.dvb.corpinter.net/FUZZ -w ~/Tools/wordlists/trickest/inventory/levels/level1.txt -config ~/.config/ffuf/ffufrc
 
cat /tmp/urls | xargs -t -I{} dirsearch -u '{}' -t 40 -r -R 3 -x 403,404,503 --random-agent -O json -o /tmp/dirsearch.json

OSINT

Metabigor

The following commands need to be executed manually:

cat tlds.txt | metabigor cert --json -o /tmp/certs.json
xargs -a tlds.txt -I{} jq -r --arg tld {} 'select(.Domain | test("\\." + $tld + "$"))' /tmp/certs.json | sponge /tmp/certs.json # Filter out of scope domains

Trích xuất org name:

cat /tmp/certs.json | jq -r '.CertInfo | match("CN=([^,]+)") | .captures[0].string' | sort -u

Trích xuất subdomains:

cat /tmp/certs.json | jq -r '.Domain' | sort -u | anew subdomains.certs.txt

Sử dụng metabigor với Org Name để tìm IP:

metabigor net -i "Org Name" --org --json -o cidr.json
 
cat cidr.json | jq -r '.CIDR' | grep -v '::' | mapcidr | anew ips.txt # Map CIDR (exclude IPv6) to IPs

Bbot

Kiểm tra output của Bbot đã chạy ở bước Passive bao gồm emails, code repository, APK, buckets, …

Dorking

Sử dụng Bug Bounty Helper.

Sử dụng uncover:

python ../misc/generate-uncover-queries.py -dL tlds.txt -o ./uncover-queries
./uncover-queries/run-uncover.sh

Engines that need to use manually:
- Hunterhow

title: FOFA
NOTE: Use favicon hashes in search results to find related assets

Resources:

• Google dorking cheatsheet: Google dork cheatsheet · GitHub

• Brave search engine: Search Operators | Brave Search

• DuckDuckGo search engine: How to use advanced syntax on DuckDuckGo Search - DuckDuckGo Help Pages

• URL Scan:

  page.domain:(domain1.com OR domain2.com)
  page.url:domain.com
  page.title:"Org Name"
  page.asnname:"Org Name"

• Dùng để search code snippet: Search Engine for Source Code - PublicWWW.com. Ví dụ: PublicWWW - PublicWWW.com.

Linux

sed -E 's/\x1B\[[0-9;]*m//g' sites.subdomains.httpx.txt # remove UNICODE chars

# 1. Stash current working changes
cd /path/to/repo && git stash push --include-untracked -m "pre-bfg cleanup"
 
# 2. Remove large folders from git history
bfg --delete-folders "{folder1,folder2,folder3}" --no-blob-protection .git
 
# 3. Clean up reflog and garbage collection
cd /path/to/repo && git reflog expire --expire=now --all && git repack -Ad && git prune --expire=now
 
# 4. Force push the cleaned history
git push --force-with-lease origin master

sed -i '' 's/^[^h]/https:\/\/&/' urls.txt # Add https to domains in a file

find . -maxdepth 1 -type f -empty -delete # Delete empty files in the current folder

find . -type f ! -name "*.*" -exec sh -c 'file "$1" | grep -q "text" && [ "$(wc -l < "$1")" -eq 1 ] && rm "$1"' _ {} \; # Remove text files that have only one line

rg -l0 'PATTERN' . | xargs -0 rm -- # Remove files in the current folder that match a pattern

jq -s 'group_by(.ip) | map(.[0]) | .[]' shodan.json | sponge shodan.json # Dedup json object in jsonl file.