```
├── .github/
├── ISSUE_TEMPLATE/
├── config.yml
├── feature_request.md
├── issue-report.md
├── dependabot.yml
├── release.yml
├── workflows/
├── build-test.yml
├── codeql-analysis.yml
├── compat-checks.yaml
├── dep-auto-merge.yml
├── dockerhub-push.yml
├── functional-test.yml
├── release-binary.yml
├── release-test.yml
├── security-crawl-maze-score.yaml
├── .gitignore
├── .goreleaser/
├── linux.yml
├── mac.yml
├── windows.yml
├── Dockerfile
├── LICENSE.md
├── Makefile
├── README.md
├── SECURITY.md
├── cmd/
├── functional-test/
├── main.go
├── run.sh
├── integration-test/
├── filters.go
├── integration-test.go
├── library.go
├── katana/
├── main.go
├── tools/
├── crawl-maze-score/
├── main.go
├── go.mod
├── go.sum
├── integration_tests/
├── run.sh
├── internal/
├── runner/
├── banner.go
├── executer.go
├── healthcheck.go
├── options.go
├── runner.go
├── testutils/
├── helper.go
├── integration.go
├── testutils.go
├── pkg/
├── engine/
├── common/
├── base.go
├── error.go
├── http.go
├── engine.go
├── hybrid/
├── crawl.go
├── doc.go
├── hijack.go
├── hybrid.go
├── parser/
├── files/
├── request.go
├── robotstxt.go
├── robotstxt_test.go
├── sitemapxml.go
├── sitemapxml_test.go
├── parser.go
├── parser_generic.go
```
## /.github/ISSUE_TEMPLATE/config.yml
```yml path="/.github/ISSUE_TEMPLATE/config.yml"
blank_issues_enabled: false
contact_links:
- name: Ask an question / advise on using katana
url: https://github.com/projectdiscovery/katana/discussions/categories/q-a
about: Ask a question or request support for using katana
- name: Share idea / feature to discuss for katana
url: https://github.com/projectdiscovery/katana/discussions/categories/ideas
about: Share idea / feature to discuss for katana
- name: Connect with PD Team (Discord)
url: https://discord.gg/projectdiscovery
about: Connect with PD Team for direct communication
```
## /.github/ISSUE_TEMPLATE/feature_request.md
---
name: Feature request
about: Request feature to implement in this project
labels: 'Type: Enhancement'
---
### Please describe your feature request:
### Describe the use case of this feature:
## /.github/ISSUE_TEMPLATE/issue-report.md
---
name: Issue report
about: Create a report to help us to improve the project
labels: 'Type: Bug'
---
### katana version:
### Current Behavior:
### Expected Behavior:
### Steps To Reproduce:
### Anything else:
## /.github/dependabot.yml
```yml path="/.github/dependabot.yml"
# To get started with Dependabot version updates, you'll need to specify which
# package ecosystems to update and where the package manifests are located.
# Please see the documentation for all configuration options:
# https://help.github.com/github/administering-a-repository/configuration-options-for-dependency-updates
version: 2
updates:
# Maintain dependencies for go modules
- package-ecosystem: "gomod"
directory: "/"
schedule:
interval: "weekly"
target-branch: "dev"
commit-message:
prefix: "chore"
include: "scope"
labels:
- "Type: Maintenance"
allow:
- dependency-name: "github.com/projectdiscovery/*"
# Maintain dependencies for GitHub Actions
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
target-branch: "dev"
commit-message:
prefix: "chore"
include: "scope"
labels:
- "Type: Maintenance"
# Maintain dependencies for docker
- package-ecosystem: "docker"
directory: "/"
schedule:
interval: "weekly"
target-branch: "dev"
commit-message:
prefix: "chore"
include: "scope"
labels:
- "Type: Maintenance"
```
## /.github/release.yml
```yml path="/.github/release.yml"
changelog:
exclude:
authors:
- app/dependabot
- dependabot
categories:
- title: 🎉 New Features
labels:
- "Type: Enhancement"
- title: 🐞 Bug Fixes
labels:
- "Type: Bug"
- title: 🔨 Maintenance
labels:
- "Type: Maintenance"
- title: Other Changes
labels:
- "*"
```
## /.github/workflows/build-test.yml
```yml path="/.github/workflows/build-test.yml"
name: 🔨 Build Test
on:
workflow_dispatch:
pull_request:
branches:
- dev
paths:
- '**.go'
- '**.mod'
jobs:
lint:
name: "Lint"
if: "${{ !endsWith(github.actor, '[bot]') }}"
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: projectdiscovery/actions/setup/go@v1
- uses: projectdiscovery/actions/golangci-lint@v1
build:
name: Test Builds
needs: [lint]
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest, windows-latest, macOS-latest]
go-version: [1.21.x]
steps:
- name: Set up Go
uses: actions/setup-go@v4
with:
go-version: ${{ matrix.go-version }}
- name: Check out code
uses: actions/checkout@v4
- name: Test
run: go test ./...
working-directory: .
- name: Build
run: go build .
working-directory: cmd/katana/
- name: Integration Tests
env:
GH_ACTION: true
run: bash run.sh
working-directory: integration_tests/
- name: Install
run: go install
working-directory: cmd/katana/
- name: Race Condition Tests
run: go build -race .
working-directory: cmd/katana/
```
## /.github/workflows/codeql-analysis.yml
```yml path="/.github/workflows/codeql-analysis.yml"
name: 🚨 CodeQL Analysis
on:
workflow_dispatch:
pull_request:
branches:
- dev
paths:
- '**.go'
- '**.mod'
jobs:
analyze:
name: Analyze
runs-on: ubuntu-latest
permissions:
actions: read
contents: read
security-events: write
strategy:
fail-fast: false
matrix:
language: [ 'go' ]
# CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python' ]
steps:
- name: Checkout repository
uses: actions/checkout@v4
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v2
with:
languages: ${{ matrix.language }}
- name: Autobuild
uses: github/codeql-action/autobuild@v2
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v2
```
## /.github/workflows/compat-checks.yaml
```yaml path="/.github/workflows/compat-checks.yaml"
name: ♾️ Compatibility Checks
on:
pull_request:
types: [opened, synchronize]
branches:
- dev
jobs:
check:
if: github.actor == 'dependabot[bot]'
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- uses: actions/checkout@v4
- uses: projectdiscovery/actions/setup/go/compat-checks@v1
with:
release-test: true
```
## /.github/workflows/dep-auto-merge.yml
```yml path="/.github/workflows/dep-auto-merge.yml"
name: 🤖 Auto Merge
on:
pull_request_review:
types: [submitted]
workflow_run:
workflows: ["♾️ Compatibility Check"]
types:
- completed
permissions:
pull-requests: write
issues: write
repository-projects: write
jobs:
auto-merge:
runs-on: ubuntu-latest
if: github.actor == 'dependabot[bot]'
steps:
- uses: actions/checkout@v4
with:
token: ${{ secrets.DEPENDABOT_PAT }}
- uses: ahmadnassri/action-dependabot-auto-merge@v2
with:
github-token: ${{ secrets.DEPENDABOT_PAT }}
target: all
```
## /.github/workflows/dockerhub-push.yml
```yml path="/.github/workflows/dockerhub-push.yml"
name: 🌥 Docker Push
on:
workflow_run:
workflows: ["🎉 Release Binary"]
types:
- completed
workflow_dispatch:
jobs:
docker:
runs-on: ubuntu-latest-16-cores
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Get Github tag
id: meta
run: |
curl --silent "https://api.github.com/repos/projectdiscovery/katana/releases/latest" | jq -r .tag_name | xargs -I {} echo TAG={} >> $GITHUB_OUTPUT
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Login to DockerHub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_TOKEN }}
- name: Build and push
uses: docker/build-push-action@v6
with:
context: .
platforms: linux/amd64,linux/arm64
push: true
tags: projectdiscovery/katana:latest,projectdiscovery/katana:${{ steps.meta.outputs.TAG }}
```
## /.github/workflows/functional-test.yml
```yml path="/.github/workflows/functional-test.yml"
name: 🧪 Functional Test
on:
pull_request:
paths:
- '**.go'
- '**.mod'
workflow_dispatch:
jobs:
functional:
name: Functional Test
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest, windows-latest, macOS-latest]
steps:
- name: Set up Go
uses: actions/setup-go@v4
with:
go-version: 1.21.x
- name: Check out code
uses: actions/checkout@v4
- name: Functional Tests
run: |
chmod +x run.sh
bash run.sh ${{ matrix.os }}
working-directory: cmd/functional-test
```
## /.github/workflows/release-binary.yml
```yml path="/.github/workflows/release-binary.yml"
name: 🎉 Release Binary
on:
push:
tags:
- v*
workflow_dispatch:
jobs:
build-mac:
runs-on: macos-latest
steps:
- name: "Check out code"
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: "Set up Go"
uses: actions/setup-go@v4
with:
go-version: 1.21.x
cache: true
- name: "Create release on GitHub"
uses: goreleaser/goreleaser-action@v4
with:
args: "release -f .goreleaser/mac.yml --clean"
version: latest
workdir: .
env:
GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
build-windows:
runs-on: windows-latest-8-cores
steps:
- name: "Check out code"
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: "Set up Go"
uses: actions/setup-go@v4
with:
go-version: 1.21.x
cache: true
- name: "Create release on GitHub"
uses: goreleaser/goreleaser-action@v4
with:
args: "release -f .goreleaser/windows.yml --clean"
version: latest
workdir: .
env:
GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
build-linux:
runs-on: ubuntu-latest-16-cores
steps:
- name: "Check out code"
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: "Set up Go"
uses: actions/setup-go@v4
with:
go-version: 1.21.x
cache: true
# todo: musl compatible?
- name: Install Dependences
run: sudo apt install gcc-aarch64-linux-gnu
- name: "Create release on GitHub"
uses: goreleaser/goreleaser-action@v4
with:
args: "release -f .goreleaser/linux.yml --clean"
version: latest
workdir: .
env:
GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
SLACK_WEBHOOK: "${{ secrets.RELEASE_SLACK_WEBHOOK }}"
DISCORD_WEBHOOK_ID: "${{ secrets.DISCORD_WEBHOOK_ID }}"
DISCORD_WEBHOOK_TOKEN: "${{ secrets.DISCORD_WEBHOOK_TOKEN }}"
```
## /.github/workflows/release-test.yml
```yml path="/.github/workflows/release-test.yml"
name: 🔨 Release Test
on:
pull_request:
paths:
- '**.yml'
- '**.go'
- '**.mod'
workflow_dispatch:
jobs:
release-test-mac:
runs-on: macos-latest
steps:
- name: "Check out code"
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Set up Go
uses: actions/setup-go@v4
with:
go-version: 1.21.x
- name: release test
uses: goreleaser/goreleaser-action@v4
with:
args: "release -f .goreleaser/mac.yml --clean --snapshot"
version: latest
workdir: .
env:
GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
release-test-linux:
runs-on: ubuntu-latest-16-cores
steps:
- name: "Check out code"
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Set up Go
uses: actions/setup-go@v4
with:
go-version: 1.21.x
# todo: musl compatible?
- name: Install Dependences
run: sudo apt update && sudo apt install gcc-aarch64-linux-gnu
- name: release test
uses: goreleaser/goreleaser-action@v4
with:
args: "release -f .goreleaser/linux.yml --clean --snapshot"
version: latest
workdir: .
env:
GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
release-test-windows:
runs-on: windows-latest-8-cores
steps:
- name: "Check out code"
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Set up Go
uses: actions/setup-go@v4
with:
go-version: 1.21.x
- name: release test
uses: goreleaser/goreleaser-action@v4
with:
args: "release -f .goreleaser/windows.yml --clean --snapshot"
version: latest
workdir: .
env:
GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
```
## /.github/workflows/security-crawl-maze-score.yaml
```yaml path="/.github/workflows/security-crawl-maze-score.yaml"
name: 🙏🏻 Security Crawl Maze Score
on:
workflow_dispatch:
jobs:
build:
name: Run Scoring
runs-on: ubuntu-latest-16-cores
steps:
- name: Set up Go
uses: actions/setup-go@v4
with:
go-version: 1.21.x
- name: Check out code
uses: actions/checkout@v4
- name: Build
run: go build .
working-directory: cmd/katana/
- name: Run Katana Standard
run: ./katana -u https://security-crawl-maze.app/ -kf all -jc -jsluice -d 10 -o output_standard.txt -cos node_modules
working-directory: cmd/katana
- name: Run Katana Headless
run: ./katana -u https://security-crawl-maze.app/ -kf all -jc -jsluice -d 10 -headless -o output_headless.txt -cos node_modules
working-directory: cmd/katana
- name: Run Score
run: go build .; ./crawl-maze-score ../../katana/output_standard.txt ../../katana/output_headless.txt
working-directory: cmd/tools/crawl-maze-score
```
## /.gitignore
```gitignore path="/.gitignore"
security-crawl-maze/
cmd/katana/katana
katana
*.exe
katana_*/
katana_*/
dist/
.vscode
.devcontainer
```
## /.goreleaser/linux.yml
```yml path="/.goreleaser/linux.yml"
env:
- GO111MODULE=on
before:
hooks:
- go mod tidy
project_name: katana
builds:
- id: katana-linux-amd64-generic
ldflags:
- -s -w
binary: katana
env:
- CGO_ENABLED=1
main: ./cmd/katana/main.go
goos:
- linux
goarch:
- amd64
- id: katana-linux-i386-generic
ldflags:
- -s -w
binary: katana
main: ./cmd/katana/main.go
goos:
- linux
goarch:
- 386
- id: katana-linux-arm
ldflags:
- -s -w
binary: katana
env:
- CGO_ENABLED=1
- CC=aarch64-linux-gnu-gcc
main: ./cmd/katana/main.go
goos:
- linux
goarch:
- arm64
archives:
- format: zip
checksum:
name_template: "{{ .ProjectName }}-linux-checksums.txt"
announce:
slack:
enabled: true
channel: '#release'
username: GoReleaser
message_template: 'New Release: {{ .ProjectName }} {{.Tag}} is published! Check it out at {{ .ReleaseURL }}'
discord:
enabled: true
message_template: '**New Release: {{ .ProjectName }} {{.Tag}}** is published! Check it out at {{ .ReleaseURL }}'
```
## /.goreleaser/mac.yml
```yml path="/.goreleaser/mac.yml"
env:
- GO111MODULE=on
before:
hooks:
- go mod tidy
project_name: katana
builds:
- id: katana-darwin
ldflags:
- -s -w
binary: katana
env:
- CGO_ENABLED=1
main: ./cmd/katana/main.go
goos:
- darwin
goarch:
- amd64
- arm64
- 386
- arm
archives:
- format: zip
name_template: '{{ .ProjectName }}_{{ .Version }}_{{ if eq .Os "darwin" }}macOS{{ else }}{{ .Os }}{{ end }}_{{ .Arch }}'
checksum:
name_template: "{{ .ProjectName }}-mac-checksums.txt"
```
## /.goreleaser/windows.yml
```yml path="/.goreleaser/windows.yml"
env:
- GO111MODULE=on
before:
hooks:
- go mod tidy
project_name: katana
builds:
- id: katana-windows
ldflags:
- -s -w
binary: katana
env:
- CGO_ENABLED=1
main: ./cmd/katana/main.go
goos:
- windows
goarch:
- 386
- arm64
- amd64
archives:
- format: zip
checksum:
name_template: "{{ .ProjectName }}-windows-checksums.txt"
```
## /Dockerfile
``` path="/Dockerfile"
FROM golang:1.23-alpine AS build-env
RUN apk add --no-cache git gcc musl-dev
WORKDIR /app
COPY . /app
RUN go mod download
RUN go build ./cmd/katana
FROM alpine:3.21.3
RUN apk add --no-cache bind-tools ca-certificates chromium
COPY --from=build-env /app/katana /usr/local/bin/
ENTRYPOINT ["katana"]
```
## /LICENSE.md
MIT License
Copyright (c) 2022 ProjectDiscovery
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
## /Makefile
``` path="/Makefile"
# Go parameters
GOCMD=go
GOBUILD=$(GOCMD) build
GOMOD=$(GOCMD) mod
GOTEST=$(GOCMD) test
GOFLAGS := -v
# This should be disabled if the binary uses pprof
LDFLAGS := -s -w
ifneq ($(shell go env GOOS),darwin)
LDFLAGS := -extldflags "-static"
endif
all: build
build:
$(GOBUILD) $(GOFLAGS) -ldflags '$(LDFLAGS)' -o "katana" cmd/katana/main.go
test:
$(GOTEST) $(GOFLAGS) ./...
integration:
cd integration_tests; bash run.sh cd ..
tidy:
$(GOMOD) tidy
```
## /README.md
A next-generation crawling and spidering framework
Features •
Installation •
Usage •
Scope •
Config •
Filters •
Join Discord
# Features
- Fast And fully configurable web crawling
- **Standard** and **Headless** mode
- **JavaScript** parsing / crawling
- Customizable **automatic form filling**
- **Scope control** - Preconfigured field / Regex
- **Customizable output** - Preconfigured fields
- INPUT - **STDIN**, **URL** and **LIST**
- OUTPUT - **STDOUT**, **FILE** and **JSON**
## Installation
katana requires Go 1.21+ to install successfully. If you encounter any installation issues, we recommend trying with the latest available version of Go, as the minimum required version may have changed. Run the command below or download a pre-compiled binary from the [release page](https://github.com/projectdiscovery/katana/releases).
```console
CGO_ENABLED=1 go install github.com/projectdiscovery/katana/cmd/katana@latest
```
**More options to install / run katana-**
Docker
> To install / update docker to latest tag -
```sh
docker pull projectdiscovery/katana:latest
```
> To run katana in standard mode using docker -
```sh
docker run projectdiscovery/katana:latest -u https://tesla.com
```
> To run katana in headless mode using docker -
```sh
docker run projectdiscovery/katana:latest -u https://tesla.com -system-chrome -headless
```
Ubuntu
> It's recommended to install the following prerequisites -
```sh
sudo apt update
sudo snap refresh
sudo apt install zip curl wget git
sudo snap install golang --classic
wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add -
sudo sh -c 'echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list'
sudo apt update
sudo apt install google-chrome-stable
```
> install katana -
```sh
go install github.com/projectdiscovery/katana/cmd/katana@latest
```
## Usage
```console
katana -h
```
This will display help for the tool. Here are all the switches it supports.
```console
Katana is a fast crawler focused on execution in automation
pipelines offering both headless and non-headless crawling.
Usage:
./katana [flags]
Flags:
INPUT:
-u, -list string[] target url / list to crawl
-resume string resume scan using resume.cfg
-e, -exclude string[] exclude host matching specified filter ('cdn', 'private-ips', cidr, ip, regex)
CONFIGURATION:
-r, -resolvers string[] list of custom resolver (file or comma separated)
-d, -depth int maximum depth to crawl (default 3)
-jc, -js-crawl enable endpoint parsing / crawling in javascript file
-jsl, -jsluice enable jsluice parsing in javascript file (memory intensive)
-ct, -crawl-duration value maximum duration to crawl the target for (s, m, h, d) (default s)
-kf, -known-files string enable crawling of known files (all,robotstxt,sitemapxml), a minimum depth of 3 is required to ensure all known files are properly crawled.
-mrs, -max-response-size int maximum response size to read (default 9223372036854775807)
-timeout int time to wait for request in seconds (default 10)
-aff, -automatic-form-fill enable automatic form filling (experimental)
-fx, -form-extraction extract form, input, textarea & select elements in jsonl output
-retry int number of times to retry the request (default 1)
-proxy string http/socks5 proxy to use
-H, -headers string[] custom header/cookie to include in all http request in header:value format (file)
-config string path to the katana configuration file
-fc, -form-config string path to custom form configuration file
-flc, -field-config string path to custom field configuration file
-s, -strategy string Visit strategy (depth-first, breadth-first) (default "depth-first")
-iqp, -ignore-query-params Ignore crawling same path with different query-param values
-tlsi, -tls-impersonate enable experimental client hello (ja3) tls randomization
-dr, -disable-redirects disable following redirects (default false)
DEBUG:
-health-check, -hc run diagnostic check up
-elog, -error-log string file to write sent requests error log
HEADLESS:
-hl, -headless enable headless hybrid crawling (experimental)
-sc, -system-chrome use local installed chrome browser instead of katana installed
-sb, -show-browser show the browser on the screen with headless mode
-ho, -headless-options string[] start headless chrome with additional options
-nos, -no-sandbox start headless chrome in --no-sandbox mode
-cdd, -chrome-data-dir string path to store chrome browser data
-scp, -system-chrome-path string use specified chrome browser for headless crawling
-noi, -no-incognito start headless chrome without incognito mode
-cwu, -chrome-ws-url string use chrome browser instance launched elsewhere with the debugger listening at this URL
-xhr, -xhr-extraction extract xhr request url,method in jsonl output
SCOPE:
-cs, -crawl-scope string[] in scope url regex to be followed by crawler
-cos, -crawl-out-scope string[] out of scope url regex to be excluded by crawler
-fs, -field-scope string pre-defined scope field (dn,rdn,fqdn) or custom regex (e.g., '(company-staging.io|company.com)') (default "rdn")
-ns, -no-scope disables host based default scope
-do, -display-out-scope display external endpoint from scoped crawling
FILTER:
-mr, -match-regex string[] regex or list of regex to match on output url (cli, file)
-fr, -filter-regex string[] regex or list of regex to filter on output url (cli, file)
-f, -field string field to display in output (url,path,fqdn,rdn,rurl,qurl,qpath,file,ufile,key,value,kv,dir,udir)
-sf, -store-field string field to store in per-host output (url,path,fqdn,rdn,rurl,qurl,qpath,file,ufile,key,value,kv,dir,udir)
-em, -extension-match string[] match output for given extension (eg, -em php,html,js)
-ef, -extension-filter string[] filter output for given extension (eg, -ef png,css)
-mdc, -match-condition string match response with dsl based condition
-fdc, -filter-condition string filter response with dsl based condition
RATE-LIMIT:
-c, -concurrency int number of concurrent fetchers to use (default 10)
-p, -parallelism int number of concurrent inputs to process (default 10)
-rd, -delay int request delay between each request in seconds
-rl, -rate-limit int maximum requests to send per second (default 150)
-rlm, -rate-limit-minute int maximum number of requests to send per minute
UPDATE:
-up, -update update katana to latest version
-duc, -disable-update-check disable automatic katana update check
OUTPUT:
-o, -output string file to write output to
-sr, -store-response store http requests/responses
-srd, -store-response-dir string store http requests/responses to custom directory
-sfd, -store-field-dir string store per-host field to custom directory
-or, -omit-raw omit raw requests/responses from jsonl output
-ob, -omit-body omit response body from jsonl output
-j, -jsonl write output in jsonl format
-nc, -no-color disable output content coloring (ANSI escape codes)
-silent display output only
-v, -verbose display verbose output
-debug display debug output
-version display project version
```
## Running Katana
### Input for katana
**katana** requires **url** or **endpoint** to crawl and accepts single or multiple inputs.
Input URL can be provided using `-u` option, and multiple values can be provided using comma-separated input, similarly **file** input is supported using `-list` option and additionally piped input (stdin) is also supported.
#### URL Input
```sh
katana -u https://tesla.com
```
#### Multiple URL Input (comma-separated)
```sh
katana -u https://tesla.com,https://google.com
```
#### List Input
```bash
$ cat url_list.txt
https://tesla.com
https://google.com
```
```
katana -list url_list.txt
```
#### STDIN (piped) Input
```sh
echo https://tesla.com | katana
```
```sh
cat domains | httpx | katana
```
Example running katana -
```console
katana -u https://youtube.com
__ __
/ /_____ _/ /____ ____ ___ _
/ '_/ _ / __/ _ / _ \/ _ /
/_/\_\\_,_/\__/\_,_/_//_/\_,_/ v0.0.1
projectdiscovery.io
[WRN] Use with caution. You are responsible for your actions.
[WRN] Developers assume no liability and are not responsible for any misuse or damage.
https://www.youtube.com/
https://www.youtube.com/about/
https://www.youtube.com/about/press/
https://www.youtube.com/about/copyright/
https://www.youtube.com/t/contact_us/
https://www.youtube.com/creators/
https://www.youtube.com/ads/
https://www.youtube.com/t/terms
https://www.youtube.com/t/privacy
https://www.youtube.com/about/policies/
https://www.youtube.com/howyoutubeworks?utm_campaign=ytgen&utm_source=ythp&utm_medium=LeftNav&utm_content=txt&u=https%3A%2F%2Fwww.youtube.com%2Fhowyoutubeworks%3Futm_source%3Dythp%26utm_medium%3DLeftNav%26utm_campaign%3Dytgen
https://www.youtube.com/new
https://m.youtube.com/
https://www.youtube.com/s/desktop/4965577f/jsbin/desktop_polymer.vflset/desktop_polymer.js
https://www.youtube.com/s/desktop/4965577f/cssbin/www-main-desktop-home-page-skeleton.css
https://www.youtube.com/s/desktop/4965577f/cssbin/www-onepick.css
https://www.youtube.com/s/_/ytmainappweb/_/ss/k=ytmainappweb.kevlar_base.0Zo5FUcPkCg.L.B1.O/am=gAE/d=0/rs=AGKMywG5nh5Qp-BGPbOaI1evhF5BVGRZGA
https://www.youtube.com/opensearch?locale=en_GB
https://www.youtube.com/manifest.webmanifest
https://www.youtube.com/s/desktop/4965577f/cssbin/www-main-desktop-watch-page-skeleton.css
https://www.youtube.com/s/desktop/4965577f/jsbin/web-animations-next-lite.min.vflset/web-animations-next-lite.min.js
https://www.youtube.com/s/desktop/4965577f/jsbin/custom-elements-es5-adapter.vflset/custom-elements-es5-adapter.js
https://www.youtube.com/s/desktop/4965577f/jsbin/webcomponents-sd.vflset/webcomponents-sd.js
https://www.youtube.com/s/desktop/4965577f/jsbin/intersection-observer.min.vflset/intersection-observer.min.js
https://www.youtube.com/s/desktop/4965577f/jsbin/scheduler.vflset/scheduler.js
https://www.youtube.com/s/desktop/4965577f/jsbin/www-i18n-constants-en_GB.vflset/www-i18n-constants.js
https://www.youtube.com/s/desktop/4965577f/jsbin/www-tampering.vflset/www-tampering.js
https://www.youtube.com/s/desktop/4965577f/jsbin/spf.vflset/spf.js
https://www.youtube.com/s/desktop/4965577f/jsbin/network.vflset/network.js
https://www.youtube.com/howyoutubeworks/
https://www.youtube.com/trends/
https://www.youtube.com/jobs/
https://www.youtube.com/kids/
```
## Crawling Mode
### Standard Mode
Standard crawling modality uses the standard go http library under the hood to handle HTTP requests/responses. This modality is much faster as it doesn't have the browser overhead. Still, it analyzes HTTP responses body as is, without any javascript or DOM rendering, potentially missing post-dom-rendered endpoints or asynchronous endpoint calls that might happen in complex web applications depending, for example, on browser-specific events.
### Headless Mode
Headless mode hooks internal headless calls to handle HTTP requests/responses directly within the browser context. This offers two advantages:
- The HTTP fingerprint (TLS and user agent) fully identify the client as a legitimate browser
- Better coverage since the endpoints are discovered analyzing the standard raw response, as in the previous modality, and also the browser-rendered one with javascript enabled.
Headless crawling is optional and can be enabled using `-headless` option.
Here are other headless CLI options -
```console
katana -h headless
Flags:
HEADLESS:
-hl, -headless enable headless hybrid crawling (experimental)
-sc, -system-chrome use local installed chrome browser instead of katana installed
-sb, -show-browser show the browser on the screen with headless mode
-ho, -headless-options string[] start headless chrome with additional options
-nos, -no-sandbox start headless chrome in --no-sandbox mode
-cdd, -chrome-data-dir string path to store chrome browser data
-scp, -system-chrome-path string use specified chrome browser for headless crawling
-noi, -no-incognito start headless chrome without incognito mode
-cwu, -chrome-ws-url string use chrome browser instance launched elsewhere with the debugger listening at this URL
-xhr, -xhr-extraction extract xhr requests
```
*`-no-sandbox`*
----
Runs headless chrome browser with **no-sandbox** option, useful when running as root user.
```console
katana -u https://tesla.com -headless -no-sandbox
```
*`-no-incognito`*
----
Runs headless chrome browser without incognito mode, useful when using the local browser.
```console
katana -u https://tesla.com -headless -no-incognito
```
*`-headless-options`*
----
When crawling in headless mode, additional chrome options can be specified using `-headless-options`, for example -
```console
katana -u https://tesla.com -headless -system-chrome -headless-options --disable-gpu,proxy-server=http://127.0.0.1:8080
```
## Scope Control
Crawling can be endless if not scoped, as such katana comes with multiple support to define the crawl scope.
*`-field-scope`*
----
Most handy option to define scope with predefined field name, `rdn` being default option for field scope.
- `rdn` - crawling scoped to root domain name and all subdomains (e.g. `*example.com`) (default)
- `fqdn` - crawling scoped to given sub(domain) (e.g. `www.example.com` or `api.example.com`)
- `dn` - crawling scoped to domain name keyword (e.g. `example`)
```console
katana -u https://tesla.com -fs dn
```
*`-crawl-scope`*
------
For advanced scope control, `-cs` option can be used that comes with **regex** support.
```console
katana -u https://tesla.com -cs login
```
For multiple in scope rules, file input with multiline string / regex can be passed.
```bash
$ cat in_scope.txt
login/
admin/
app/
wordpress/
```
```console
katana -u https://tesla.com -cs in_scope.txt
```
*`-crawl-out-scope`*
-----
For defining what not to crawl, `-cos` option can be used and also support **regex** input.
```console
katana -u https://tesla.com -cos logout
```
For multiple out of scope rules, file input with multiline string / regex can be passed.
```bash
$ cat out_of_scope.txt
/logout
/log_out
```
```console
katana -u https://tesla.com -cos out_of_scope.txt
```
*`-no-scope`*
----
Katana is default to scope `*.domain`, to disable this `-ns` option can be used and also to crawl the internet.
```console
katana -u https://tesla.com -ns
```
*`-display-out-scope`*
----
As default, when scope option is used, it also applies for the links to display as output, as such **external URLs are default to exclude** and to overwrite this behavior, `-do` option can be used to display all the external URLs that exist in targets scoped URL / Endpoint.
```
katana -u https://tesla.com -do
```
Here is all the CLI options for the scope control -
```console
katana -h scope
Flags:
SCOPE:
-cs, -crawl-scope string[] in scope url regex to be followed by crawler
-cos, -crawl-out-scope string[] out of scope url regex to be excluded by crawler
-fs, -field-scope string pre-defined scope field (dn,rdn,fqdn) (default "rdn")
-ns, -no-scope disables host based default scope
-do, -display-out-scope display external endpoint from scoped crawling
```
## Crawler Configuration
Katana comes with multiple options to configure and control the crawl as the way we want.
*`-depth`*
----
Option to define the `depth` to follow the urls for crawling, the more depth the more number of endpoint being crawled + time for crawl.
```
katana -u https://tesla.com -d 5
```
*`-js-crawl`*
----
Option to enable JavaScript file parsing + crawling the endpoints discovered in JavaScript files, disabled as default.
```
katana -u https://tesla.com -jc
```
*`-crawl-duration`*
----
Option to predefined crawl duration, disabled as default.
```
katana -u https://tesla.com -ct 2
```
*`-known-files`*
----
Option to enable crawling `robots.txt` and `sitemap.xml` file, disabled as default.
```
katana -u https://tesla.com -kf robotstxt,sitemapxml
```
*`-automatic-form-fill`*
----
Option to enable automatic form filling for known / unknown fields, known field values can be customized as needed by updating form config file at `$HOME/.config/katana/form-config.yaml`.
Automatic form filling is experimental feature.
```
katana -u https://tesla.com -aff
```
## Authenticated Crawling
Authenticated crawling involves including custom headers or cookies in HTTP requests to access protected resources. These headers provide authentication or authorization information, allowing you to crawl authenticated content / endpoint. You can specify headers directly in the command line or provide them as a file with katana to perform authenticated crawling.
> **Note**: User needs to be manually perform the authentication and export the session cookie / header to file to use with katana.
*`-headers`*
----
Option to add a custom header or cookie to the request.
> Syntax of [headers](https://datatracker.ietf.org/doc/html/rfc7230#section-3.2) in the HTTP specification
Here is an example of adding a cookie to the request:
```
katana -u https://tesla.com -H 'Cookie: usrsess=AmljNrESo'
```
It is also possible to supply headers or cookies as a file. For example:
```
$ cat cookie.txt
Cookie: PHPSESSIONID=XXXXXXXXX
X-API-KEY: XXXXX
TOKEN=XX
```
```
katana -u https://tesla.com -H cookie.txt
```
There are more options to configure when needed, here is all the config related CLI options -
```console
katana -h config
Flags:
CONFIGURATION:
-r, -resolvers string[] list of custom resolver (file or comma separated)
-d, -depth int maximum depth to crawl (default 3)
-jc, -js-crawl enable endpoint parsing / crawling in javascript file
-ct, -crawl-duration int maximum duration to crawl the target for
-kf, -known-files string enable crawling of known files (all,robotstxt,sitemapxml)
-mrs, -max-response-size int maximum response size to read (default 9223372036854775807)
-timeout int time to wait for request in seconds (default 10)
-aff, -automatic-form-fill enable automatic form filling (experimental)
-fx, -form-extraction enable extraction of form, input, textarea & select elements
-retry int number of times to retry the request (default 1)
-proxy string http/socks5 proxy to use
-H, -headers string[] custom header/cookie to include in request
-config string path to the katana configuration file
-fc, -form-config string path to custom form configuration file
-flc, -field-config string path to custom field configuration file
-s, -strategy string Visit strategy (depth-first, breadth-first) (default "depth-first")
```
### Connecting to Active Browser Session
Katana can also connect to active browser session where user is already logged in and authenticated. and use it for crawling. The only requirement for this is to start browser with remote debugging enabled.
Here is an example of starting chrome browser with remote debugging enabled and using it with katana -
**step 1) First Locate path of chrome executable**
| Operating System | Chromium Executable Location | Google Chrome Executable Location |
|------------------|------------------------------|-----------------------------------|
| Windows (64-bit) | `C:\Program Files (x86)\Google\Chromium\Application\chrome.exe` | `C:\Program Files (x86)\Google\Chrome\Application\chrome.exe` |
| Windows (32-bit) | `C:\Program Files\Google\Chromium\Application\chrome.exe` | `C:\Program Files\Google\Chrome\Application\chrome.exe` |
| macOS | `/Applications/Chromium.app/Contents/MacOS/Chromium` | `/Applications/Google Chrome.app/Contents/MacOS/Google Chrome` |
| Linux | `/usr/bin/chromium` | `/usr/bin/google-chrome` |
**step 2) Start chrome with remote debugging enabled and it will return websocker url. For example, on MacOS, you can start chrome with remote debugging enabled using following command** -
```console
$ /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --remote-debugging-port=9222
DevTools listening on ws://127.0.0.1:9222/devtools/browser/c5316c9c-19d6-42dc-847a-41d1aeebf7d6
```
> Now login to the website you want to crawl and keep the browser open.
**step 3) Now use the websocket url with katana to connect to the active browser session and crawl the website**
```console
katana -headless -u https://tesla.com -cwu ws://127.0.0.1:9222/devtools/browser/c5316c9c-19d6-42dc-847a-41d1aeebf7d6 -no-incognito
```
> **Note**: you can use `-cdd` option to specify custom chrome data directory to store browser data and cookies but that does not save session data if cookie is set to `Session` only or expires after certain time.
## Filters
*`-field`*
----
Katana comes with built in fields that can be used to filter the output for the desired information, `-f` option can be used to specify any of the available fields.
```
-f, -field string field to display in output (url,path,fqdn,rdn,rurl,qurl,qpath,file,key,value,kv,dir,udir)
```
Here is a table with examples of each field and expected output when used -
| FIELD | DESCRIPTION | EXAMPLE |
| ------- | --------------------------- | ------------------------------------------------------------ |
| `url` | URL Endpoint | `https://admin.projectdiscovery.io/admin/login?user=admin&password=admin` |
| `qurl` | URL including query param | `https://admin.projectdiscovery.io/admin/login.php?user=admin&password=admin` |
| `qpath` | Path including query param | `/login?user=admin&password=admin` |
| `path` | URL Path | `https://admin.projectdiscovery.io/admin/login` |
| `fqdn` | Fully Qualified Domain name | `admin.projectdiscovery.io` |
| `rdn` | Root Domain name | `projectdiscovery.io` |
| `rurl` | Root URL | `https://admin.projectdiscovery.io` |
| `ufile` | URL with File | `https://admin.projectdiscovery.io/login.js` |
| `file` | Filename in URL | `login.php` |
| `key` | Parameter keys in URL | `user,password` |
| `value` | Parameter values in URL | `admin,admin` |
| `kv` | Keys=Values in URL | `user=admin&password=admin` |
| `dir` | URL Directory name | `/admin/` |
| `udir` | URL with Directory | `https://admin.projectdiscovery.io/admin/` |
Here is an example of using field option to only display all the urls with query parameter in it -
```
katana -u https://tesla.com -f qurl -silent
https://shop.tesla.com/en_au?redirect=no
https://shop.tesla.com/en_nz?redirect=no
https://shop.tesla.com/product/men_s-raven-lightweight-zip-up-bomber-jacket?sku=1740250-00-A
https://shop.tesla.com/product/tesla-shop-gift-card?sku=1767247-00-A
https://shop.tesla.com/product/men_s-chill-crew-neck-sweatshirt?sku=1740176-00-A
https://www.tesla.com/about?redirect=no
https://www.tesla.com/about/legal?redirect=no
https://www.tesla.com/findus/list?redirect=no
```
### Custom Fields
You can create custom fields to extract and store specific information from page responses using regex rules. These custom fields are defined using a YAML config file and are loaded from the default location at `$HOME/.config/katana/field-config.yaml`. Alternatively, you can use the `-flc` option to load a custom field config file from a different location.
Here is example custom field.
```yaml
- name: email
type: regex
regex:
- '([a-zA-Z0-9._-]+@[a-zA-Z0-9._-]+\.[a-zA-Z0-9_-]+)'
- '([a-zA-Z0-9+._-]+@[a-zA-Z0-9._-]+\.[a-zA-Z0-9_-]+)'
- name: phone
type: regex
regex:
- '\d{3}-\d{8}|\d{4}-\d{7}'
```
When defining custom fields, following attributes are supported:
- **name** (required)
> The value of **name** attribute is used as the `-field` cli option value.
- **type** (required)
> The type of custom attribute, currently supported option - `regex`
- **part** (optional)
> The part of the response to extract the information from. The default value is `response`, which includes both the header and body. Other possible values are `header` and `body`.
- group (optional)
> You can use this attribute to select a specific matched group in regex, for example: `group: 1`
#### Running katana using custom field:
```console
katana -u https://tesla.com -f email,phone
```
*`-store-field`*
---
To compliment `field` option which is useful to filter output at run time, there is `-sf, -store-fields` option which works exactly like field option except instead of filtering, it stores all the information on the disk under `katana_field` directory sorted by target url. Use `-sfd` or `-store-field-dir` to store data in a different location.
```
katana -u https://tesla.com -sf key,fqdn,qurl -silent
```
```bash
$ ls katana_field/
https_www.tesla.com_fqdn.txt
https_www.tesla.com_key.txt
https_www.tesla.com_qurl.txt
```
The `-store-field` option can be useful for collecting information to build a targeted wordlist for various purposes, including but not limited to:
- Identifying the most commonly used parameters
- Discovering frequently used paths
- Finding commonly used files
- Identifying related or unknown subdomains
### Katana Filters
*`-extension-match`*
---
Crawl output can be easily matched for specific extension using `-em` option to ensure to display only output containing given extension.
```
katana -u https://tesla.com -silent -em js,jsp,json
```
*`-extension-filter`*
---
Crawl output can be easily filtered for specific extension using `-ef` option which ensure to remove all the urls containing given extension.
```
katana -u https://tesla.com -silent -ef css,txt,md
```
*`-match-regex`*
---
The `-match-regex` or `-mr` flag allows you to filter output URLs using regular expressions. When using this flag, only URLs that match the specified regular expression will be printed in the output.
```
katana -u https://tesla.com -mr 'https://shop\.tesla\.com/*' -silent
```
*`-filter-regex`*
---
The `-filter-regex` or `-fr` flag allows you to filter output URLs using regular expressions. When using this flag, it will skip the URLs that are match the specified regular expression.
```
katana -u https://tesla.com -fr 'https://www\.tesla\.com/*' -silent
```
### Advance Filtering
Katana supports DSL-based expressions for advanced matching and filtering capabilities:
- To match endpoints with a 200 status code:
```shell
katana -u https://www.hackerone.com -mdc 'status_code == 200'
```
- To match endpoints that contain "default" and have a status code other than 403:
```shell
katana -u https://www.hackerone.com -mdc 'contains(endpoint, "default") && status_code != 403'
```
- To match endpoints with PHP technologies:
```shell
katana -u https://www.hackerone.com -mdc 'contains(to_lower(technologies), "php")'
```
- To filter out endpoints running on Cloudflare:
```shell
katana -u https://www.hackerone.com -fdc 'contains(to_lower(technologies), "cloudflare")'
```
DSL functions can be applied to any keys in the jsonl output. For more information on available DSL functions, please visit the [dsl project](https://github.com/projectdiscovery/dsl).
Here are additional filter options -
```console
katana -h filter
Flags:
FILTER:
-mr, -match-regex string[] regex or list of regex to match on output url (cli, file)
-fr, -filter-regex string[] regex or list of regex to filter on output url (cli, file)
-f, -field string field to display in output (url,path,fqdn,rdn,rurl,qurl,qpath,file,ufile,key,value,kv,dir,udir)
-sf, -store-field string field to store in per-host output (url,path,fqdn,rdn,rurl,qurl,qpath,file,ufile,key,value,kv,dir,udir)
-em, -extension-match string[] match output for given extension (eg, -em php,html,js)
-ef, -extension-filter string[] filter output for given extension (eg, -ef png,css)
-mdc, -match-condition string match response with dsl based condition
-fdc, -filter-condition string filter response with dsl based condition
```
## Rate Limit
It's easy to get blocked / banned while crawling if not following target websites limits, katana comes with multiple option to tune the crawl to go as fast / slow we want.
*`-delay`*
-----
option to introduce a delay in seconds between each new request katana makes while crawling, disabled as default.
```
katana -u https://tesla.com -delay 20
```
*`-concurrency`*
-----
option to control the number of urls per target to fetch at the same time.
```
katana -u https://tesla.com -c 20
```
*`-parallelism`*
-----
option to define number of target to process at same time from list input.
```
katana -u https://tesla.com -p 20
```
*`-rate-limit`*
-----
option to use to define max number of request can go out per second.
```
katana -u https://tesla.com -rl 100
```
*`-rate-limit-minute`*
-----
option to use to define max number of request can go out per minute.
```
katana -u https://tesla.com -rlm 500
```
Here is all long / short CLI options for rate limit control -
```console
katana -h rate-limit
Flags:
RATE-LIMIT:
-c, -concurrency int number of concurrent fetchers to use (default 10)
-p, -parallelism int number of concurrent inputs to process (default 10)
-rd, -delay int request delay between each request in seconds
-rl, -rate-limit int maximum requests to send per second (default 150)
-rlm, -rate-limit-minute int maximum number of requests to send per minute
```
## Output
Katana support both file output in plain text format as well as JSON which includes additional information like, `source`, `tag`, and `attribute` name to co-related the discovered endpoint.
*`-output`*
By default, katana outputs the crawled endpoints in plain text format. The results can be written to a file by using the -output option.
```console
katana -u https://example.com -no-scope -output example_endpoints.txt
```
*`-jsonl`*
---
```console
katana -u https://example.com -jsonl | jq .
```
```json
{
"timestamp": "2023-03-20T16:23:58.027559+05:30",
"request": {
"method": "GET",
"endpoint": "https://example.com",
"raw": "GET / HTTP/1.1\r\nHost: example.com\r\nUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36\r\nAccept-Encoding: gzip\r\n\r\n"
},
"response": {
"status_code": 200,
"headers": {
"accept_ranges": "bytes",
"expires": "Mon, 27 Mar 2023 10:53:58 GMT",
"last_modified": "Thu, 17 Oct 2019 07:18:26 GMT",
"content_type": "text/html; charset=UTF-8",
"server": "ECS (dcb/7EA3)",
"vary": "Accept-Encoding",
"etag": "\"3147526947\"",
"cache_control": "max-age=604800",
"x_cache": "HIT",
"date": "Mon, 20 Mar 2023 10:53:58 GMT",
"age": "331239"
},
"body": "\n\n\n Example Domain\n\n \n \n \n \n\n\n\n\n
Example Domain
\n
This domain is for use in illustrative examples in documents. You may use this\n domain in literature without prior coordination or asking for permission.
\n
More information...
\n
\n
Example Domain
\n
This domain is for use in illustrative examples in documents. You may use this\n domain in literature without prior coordination or asking for permission.
\n
More information...
\n
katana is made with ❤️ by the [projectdiscovery](https://projectdiscovery.io) team and distributed under [MIT License](LICENSE.md).
