545 Commits

Author SHA1 Message Date
DoI
f520d7e843
XPath matcher support (#4087)
* Added xpath response matching support

* Add validation for user-supplied XPath

* xpath matcher comment fix

* Added XPath matched documentation

* minor changes: remove warnings

---------

Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
2023-08-25 22:41:51 +05:30
Tarun Koyalwar
b4b769d501 fix logic bug in request[s] condition 2023-08-18 02:37:35 +05:30
Ramana Reddy
7997e8dbec
Fix edge cases disable-path-automerge (#4035)
* fix edge cases for disable-path-automerge

* misc update
2023-08-10 19:28:05 +05:30
Dogan Can Bakir
0776b2e237
fix curl command (#4014) 2023-08-02 00:47:42 +05:30
Josh Soref
4c1c5301b9
Spelling (#4008)
* spelling: addresses

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: asynchronous

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: basic

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: brute force

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: constant

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: disables

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: engine

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: every time

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: execution

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: false positives

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: from

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: further

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: github

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: gitlab

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: highlight

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: hygiene

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: ignore

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: input

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: item

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: itself

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: latestxxx

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: navigation

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: negative

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: nonexistent

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: occurred

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: override

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: overrides

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: payload

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: performed

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: respective

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: retrieve

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: scanlist

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: separated

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: separator

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: severity

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: source

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: strategy

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: string

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: templates

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: terminal

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: timeout

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: trailing slash

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: trailing

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: websocket

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

---------

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
2023-08-02 00:03:43 +05:30
Tarun Koyalwar
beb1bf6d2c
headless: automerge and other improvements (#3958)
* headless: automerge and other improvements

* fix typo in function signature
2023-07-28 19:28:20 +05:30
Mzack9999
e5154d362a
fixing payload load (#3927)
* fixing payload load

* Added tests for load payloads edge-case + fixed error

* Added separate flags for network and file sandbox

* Fixed tests for payload loader

* Fixed integration tests locally

* readme update

---------

Co-authored-by: Ice3man <nizamulrana@gmail.com>
Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-07-14 19:39:32 +05:30
Tarun Koyalwar
d51e058791
add ErrNoMoreRequests for generator (#3918)
* add ErrNoMoreRequests for generator

* fix gh repo name convention

* fix dirname in unit test
2023-07-13 00:51:06 +05:30
Ramana Reddy
1eb4c7c80c
support disable-path-automerge in unsafe mode (#3888)
* support disable-path-automerge in unsafe mode

* fix dump request url

* dep update

---------

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-07-04 19:04:13 +05:30
Tarun Koyalwar
38129bac18
preserve order of query parameters (#3887)
* preserve order of parameters

* rawhttp version bump

---------

Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
2023-07-03 12:43:24 +05:30
Mzack9999
c9d0942bc1
Extend headless contextargs (#3850)
* extend headless contextargs

* using darwin-latest

* grouping page options

* temp commenting code out

* fixing test

* adding more checks

* more checks

* fixing first navigation metadata

* adding integration test

* proto update

---------

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-06-26 22:55:51 +05:30
Mzack9999
fa199ed3b3
Improving clientpool with client certificates (#3851)
* Improving clientpool with client certificates

* adding test case

* Revert "Merge branch 'dev' into issue-3800-client-cert"

This reverts commit 7f057d742f4b9bda8e83b2052e29617b86b6776d, reversing
changes made to 7297cebcf8bb0f88961b644fc2ac7c040df8ffd9.

* Revert "Revert "Merge branch 'dev' into issue-3800-client-cert""

This reverts commit 2053a248a0cdc2002e0b4b4faa3472cf11c29760.

* go fmt

---------

Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-06-26 17:45:12 +05:30
Mzack9999
2a32ed9cba
Adding random tls impersonate (#3844)
* adding random tls impersonate

* dep update

---------

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-06-21 17:17:18 +05:30
Sandeep Singh
0c8ec5e535
fix output path in unsafe mode (#3831)
Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
2023-06-20 01:25:22 +05:30
Ramana Reddy
cddae989f3
Add template option to disable merging target url path with raw request path (#3799)
* add template option to disable merging target url path with raw request path

* rename disable-merge-path -> disable-path-automerge
add integration test
2023-06-19 20:22:17 +05:30
Shubham Rasal
a34b94e62f
Issue 3339 headless fuzz (#3790)
* Basic headless fuzzing

* Remove debug statements

* Add integration tests

* Update template

* Fix recognize payload value in matcher

* Update tempalte

* use req.SetURL()

---------

Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
2023-06-09 05:50:44 +05:30
Keith Chason
4d6080f3bc
"Executer" to "Executor" (#3760)
* Fix spelling of "executer" to "executor"

* minor change: use defer file.Close()

---------

Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
2023-06-01 02:28:10 +05:30
Mzack9999
0d2d510689
Adding support for constants (#3692)
* adding support for constants

* fixing typo

* adding integration test

* fixing lint issues

* fixing template syntax
2023-05-25 22:02:35 +05:30
Shubham Rasal
449afc0c5c
Issue 3564 var override (#3599)
* Check if the variables are override by other means

- you can override the template variable value using command line flags

* Update lazy eval logic

- previously, we were checking any function/expression in variable
- now, update the logic, lazy eval only if variable contains any
  protocol variable(global)

* add integration tests

* Add test to check the dsl function working in variable

* gather all generate variables logic in utils

* go mod update

* Refactor the generate variables function

* go mod update+ fix typo

---------

Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
2023-05-02 23:49:56 +05:30
Tarun Koyalwar
7f5e4e2336
aws signer: fix missing x-content-sha256 header (#3601)
* fix missing x-content-sha256 header

* fix variable priority in self-contained templates

* remove debug statement

* adds generic raw request parser for self-contained req

* more integration tests

* bug fix: 10x faster race requests

* fix failing integration test
2023-05-01 12:15:35 +05:30
Tarun Koyalwar
4e6ef4490e
duplicated params in self contained requests (#3608)
* fix duplicated params in self-contained+ export extracted values to file

* add integration tests + fix percentage overflow in pb

* fix integration test template id

* integration test: validate if file exists
2023-04-26 12:35:07 +05:30
Tarun Koyalwar
bf08913cd0
update logic + config management refactor (#3567)
* adds template manager

* refactor: checkpoint

* centrailized config & template download logic

* refactor removed unused code

* use global template directory

* update related bug fixes

* bug fix create cfg dir if missing

* fix lint error

* bug fix skip writing template dir in callback

* misc update

* remove unused code

* use strings.equalfold for comparison

---------

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-04-19 21:58:48 +05:30
Shubham Rasal
d0f22f8b73
Issue 3488 http race (#3533)
* debug

* Add body after the request creation

* fix race_count template hangs

* remove printf

* update if condition
2023-04-19 01:57:53 +05:30
Mzack9999
6f4b1ae48a
Replacing ccache with generic gcache (#3523)
* Replacing ccache with generic gcache

* fixing lint issues

* removing unecessary hashing + using errorutils

* making test more tolerant

* removing dead code + refactor

* removing redundant code

* removing race

* maint

* moving code

* adding more iterations

* note + typo

* temporary fixing stop-at-first-match with interact

* wrapping internal map with mux

* sort before running integration test

* fix deadlock in requestShouldStopAtFirstMatch

* add timeout to integration_test workflow

* attempting to remove outer lock

* adds interactsh protocol tests in integration_test

---------

Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
2023-04-16 23:19:35 +05:30
Ramana Reddy
6ffdfcf19c
fix rate-limit on query fuzzing (#3458) (#3532) 2023-04-12 23:55:21 +05:30
Shubham Rasal
45cc676f96
Evaluate payload variables (#3503)
* Evaluate payload variables

* Add variables evaluation

* Extend variables test

- to check evaluation of global variables in variables
- to check evaluation of golbal variables in payload

* Add default and cli variables to websocket, whois and dns proto

- use url.Parse with urlutil.Parse
2023-04-12 01:50:58 +05:30
Mzack9999
4c0d988a67 reworking interact mutex mechanism 2023-03-17 14:41:16 +01:00
Tarun Koyalwar
c3771e874d
fix data race in internal resultevent (#3432) 2023-03-16 23:20:38 +05:30
Ramana Reddy
c9634fae72
Issue 3350 matcher condition or not work (#3397)
* fix or condition match even interactsh includes as matcher-part (#3350)

* add integration test

* add new template to integration test

* matcher-condtion: test case for both conditions

* fix lint errors

* upgrade dependencies

---------

Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
2023-03-15 20:45:44 +05:30
Shubham Rasal
572c8eb780
Issue 2987 fuzz options (#3355)
* Add override fuzzing type and mode flags

* Update english readme

* Fix failing tests

* Add the integration tests

- validate the command line overriding type and mode for fuzzing
2023-03-06 16:56:38 +05:30
Tarun Koyalwar
d9e953acfa
fix file input in custom vars for self contained http template (#3385)
* fix file input in variables(-V)

* fix lint error

* fix nuclei-ignore file failures
2023-03-04 04:57:27 +05:30
Mzack9999
d80cbef51d
Merge pull request #3333 from CodFrm/main
fix some json deserialization issues
2023-02-28 09:02:04 +01:00
王一之
27fefe59d3 fix json deserialization issues 2023-02-27 14:29:49 +08:00
Tarun Koyalwar
8cdc1338fc adds missing json tags and unit test 2023-02-22 02:24:45 +05:30
Tarun Koyalwar
21b03a2e8a
bug fix in url path and adds integration tests (#3331)
* fix unsafe edgecases+ adds integration test

* bug fixes and more url testcases

* upgrade cfssl

* fix template id in integration test
2023-02-20 22:26:04 +05:30
王一之
85090b7531 fix some json deserialization issues 2023-02-17 14:21:25 +08:00
Mzack9999
d608ffaeb2
clear after stop (#3312)
* clear after stop

* fixing data races

* adding atomic cache

* fixing lint errors

* fixing imports
2023-02-13 16:46:41 +05:30
Tarun Koyalwar
e622b989fe
fix url re-encoding issues (#3294)
* fix double url encoding in urls

* remove extra slash

* url encode matchedURL
2023-02-10 18:28:28 +05:30
Mzack9999
7556416e5b
adding interactsh support to sni (#3276) 2023-02-07 14:02:10 +05:30
Tarun Koyalwar
32a6adb82e
fix data race in payload (#3265) 2023-02-05 00:00:01 +05:30
Tarun Koyalwar
628628893c
fix make http request inconsistencies (#3243)
* fix make http request inconsistencies

* remove parameters from http vars

* fix trailingslash unit test

* naming conventions: best practices

* fix publish docs action

* remove branch ref from push
2023-02-01 17:23:28 +05:30
Tarun Koyalwar
0b2a3e296a
fix url encoding issues and inconsistencies (#3211)
* fix url encoding issues

* complete requested changes and improvements

* fix missing issue-tracker-config.yaml

* fuzz: deepcopy and use urlutil.Params
2023-01-24 22:04:52 +05:30
Tarun Koyalwar
edcab07fec
fix aws signer missing template variables (#3206)
* aws sign: fix missing variables

* signer: add aws defaults

* aws signer default values
2023-01-24 20:50:20 +05:30
Tarun Koyalwar
6ebf5a789e
fix host spray race condition (#3213)
* core: bug fixes

* best practices: uniform comments
2023-01-20 23:49:04 +05:30
Tarun Koyalwar
f646e00c3d
fix unsafe raw request matchedUrl (#3155)
* fix unsafe raw request matchedurl

* quote metadata in CLI output
2023-01-05 21:02:36 +05:30
Tarun Koyalwar
4aa2002e72
urlencode key characters only (#3150)
* only encode key characters

* improve test cases
2023-01-05 16:41:59 +05:30
Tarun Koyalwar
e66ed30cec
fix missing trailing slash (#3127)
* raw: fix missing trailing slash

* adds rawpath integration test

* rename trailing slash test
2023-01-03 23:45:34 +05:30
Tarun Koyalwar
a5b39dcaa5
automerge url parameters from input and templates (#3010)
* fix automerge url parameters

* fix url encoding & refactor raw request

* handle trailing slash edgecases

* minor code refactoring
2022-12-13 12:09:31 +05:30
Mzack9999
96c1dd3720
Adding custom ip to protocol generated variables (#3011)
* lint errors

* Extending context args support

* Ip => ip
2022-12-10 00:17:03 +05:30
Ice3man
30f26a32ed Do not consider fuzzing template during clustering 2022-12-05 23:07:38 +05:30