45 Commits

Author SHA1 Message Date
Daniil Morozov
c377221a78
header fuzzing support in http templates (#4114)
* Add headersPartType for fuzzing

* fix nil pointer dereference for headless mode

* minor changes+ add integration test

* update template in fuzz-header-multiple

---------

Co-authored-by: 0x123456789 <0x123456789>
Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
2023-09-19 00:01:32 +05:30
Sandeep Singh
d3928e080d
optional file read in headless protocol (#4055)
* use -lfa and -lna in headless

* fix lna in headless

* misc update

* fix nil pointer dereference in test

* fix lint & unit test

* use urlutil

* headless protocol scheme improvements

* add unit and integration tests

* run unit test from binary

---------

Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
2023-08-25 18:30:46 +05:30
Josh Soref
4c1c5301b9
Spelling (#4008)
* spelling: addresses

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: asynchronous

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: basic

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: brute force

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: constant

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: disables

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: engine

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: every time

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: execution

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: false positives

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: from

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: further

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: github

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: gitlab

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: highlight

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: hygiene

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: ignore

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: input

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: item

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: itself

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: latestxxx

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: navigation

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: negative

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: nonexistent

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: occurred

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: override

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: overrides

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: payload

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: performed

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: respective

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: retrieve

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: scanlist

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: separated

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: separator

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: severity

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: source

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: strategy

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: string

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: templates

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: terminal

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: timeout

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: trailing slash

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: trailing

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

* spelling: websocket

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

---------

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
2023-08-02 00:03:43 +05:30
Tarun Koyalwar
beb1bf6d2c
headless: automerge and other improvements (#3958)
* headless: automerge and other improvements

* fix typo in function signature
2023-07-28 19:28:20 +05:30
Tarun Koyalwar
d51e058791
add ErrNoMoreRequests for generator (#3918)
* add ErrNoMoreRequests for generator

* fix gh repo name convention

* fix dirname in unit test
2023-07-13 00:51:06 +05:30
Mzack9999
c9d0942bc1
Extend headless contextargs (#3850)
* extend headless contextargs

* using darwin-latest

* grouping page options

* temp commenting code out

* fixing test

* adding more checks

* more checks

* fixing first navigation metadata

* adding integration test

* proto update

---------

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-06-26 22:55:51 +05:30
Dogan Can Bakir
a4ca2021cd
Add headless header and status matchers (#3794)
* add headless header and status matchers

* rename headers as header

* add integration test for header+status

* fix typo
2023-06-09 15:03:03 +05:30
Shubham Rasal
a34b94e62f
Issue 3339 headless fuzz (#3790)
* Basic headless fuzzing

* Remove debug statements

* Add integration tests

* Update template

* Fix recognize payload value in matcher

* Update tempalte

* use req.SetURL()

---------

Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
2023-06-09 05:50:44 +05:30
Keith Chason
4d6080f3bc
"Executer" to "Executor" (#3760)
* Fix spelling of "executer" to "executor"

* minor change: use defer file.Close()

---------

Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
2023-06-01 02:28:10 +05:30
Mzack9999
0d2d510689
Adding support for constants (#3692)
* adding support for constants

* fixing typo

* adding integration test

* fixing lint issues

* fixing template syntax
2023-05-25 22:02:35 +05:30
Shubham Rasal
449afc0c5c
Issue 3564 var override (#3599)
* Check if the variables are override by other means

- you can override the template variable value using command line flags

* Update lazy eval logic

- previously, we were checking any function/expression in variable
- now, update the logic, lazy eval only if variable contains any
  protocol variable(global)

* add integration tests

* Add test to check the dsl function working in variable

* gather all generate variables logic in utils

* go mod update

* Refactor the generate variables function

* go mod update+ fix typo

---------

Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
2023-05-02 23:49:56 +05:30
Mzack9999
a96f764959
extending request/response hijacking with native calls (#3091)
* extending request/response hijacking with native calls

* fixing tests
2023-01-05 17:26:18 +05:30
Mzack9999
093d691c16
Issue 3033 deny list (#3037)
* fixing file deny list + refactoring

* err variable renaming

* removing redundant function

* removing unused code

* adding check on empty operator

* updating tests
2022-12-21 02:29:28 +05:30
xm1k3
aaae68ee50
The debug flags do not work correctly with headless templates (#3030)
* added check on BaseURL

* added debug flag instead on info

* added also debug response flag

* code refactoring
2022-12-20 17:30:34 +05:30
Tarun Koyalwar
63150a8ee4
add stop-at-first-match for headless protocol (#3034)
* headless: adds stop-at-first-match

* GH workflow fix

* misc fix

* headless use spm option

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2022-12-19 19:52:17 +05:30
Mzack9999
96c1dd3720
Adding custom ip to protocol generated variables (#3011)
* lint errors

* Extending context args support

* Ip => ip
2022-12-10 00:17:03 +05:30
Mzack9999
1fbbca66f9
Adding support to scan all v4/v6 IPs (#2709)
* Adding support to scan all v4/v6 IPs

* adding tests

* metainput prototype

* using new signature

* fixing nil pointer

* adding request context with metadata

* removing log instruction

* fixing merge conflicts

* adding clone helpers

* attempting to fix ipv6 square parenthesis wrap

* fixing dialed ip info

* fixing syntax

* fixing output ip selection

* adding integration tests

* disabling test due to gh ipv6 issue

* using ipv4 only due to GH limited networking

* extending metainput marshaling

* fixing hmap key

* adding test for httpx integration

* fixing lint error

* reworking marshaling/id-calculation

* adding ip version validation

* improving handling non url targets

* fixing condition check
2022-11-09 18:48:56 +05:30
Ice3man
b9472cf7e1
Added fuzzing support for query params + var dump feature (#2679)
* Added fuzzing support for query params + var dump feature

* Added query-fuzz integration test

* Fixed payloads + added keys-regex fuzz parameter

* Fixed interactsh not working + misc

* Fixed evaluation + added global variables/dsl support to payloads

* Misc fixes related to variables evaluations

* Added http variables support to fuzz

* misc

* Misc

* Added testing playground + misc renaming

* Added support for path and raw request to fuzzing

* Fixed fuzz integration test

* Fixed variable unresolved issue

* Add multiple parameter support with same name

* Added parameter value as 'value' dsl variable for parts

Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
2022-11-01 20:28:50 +05:30
Mzack9999
781e4e6105
Shared Execution Context Prototype (#2576)
* renaming var

* Introducing shared execution context prototype

* fixing field name

* adding shared values propagation

* adding shared context lock

* add slice values normalization

* adding integration tests

* adding metadata support for dns

* adding multi-protocol context sharing test

* removing debug test files

* moving contextargs around

* adding comments

* refactoring code

- getter/setter for complex types
- using pointers to avoid heap allocations
2022-10-03 15:42:20 +05:30
Ice3man
0be596efb4
Added variable debug support with debug mode (#2442)
* Added variable debug support with debug mode

* Added changes as per review comments

* Fixed debug request condition
2022-08-25 15:37:03 +05:30
Mzack9999
02eaf91e6a
Adding variables support for headless templates (#2064) 2022-05-27 21:31:56 +05:30
Sami
6ca4374f91
sonar category: String literals should not be duplicated (#1944)
* sonar category: String literals should not be duplicated

* lint error fix

* better naming conventions for constants

* improved naming conventions and methods
2022-05-12 15:40:14 +05:30
Ice3man
d0d65f8d6b Added integration tests for variables + misc changes 2022-04-02 02:14:00 +05:30
Ice3man
17fa23e737 Added variable introduction to headless 2022-04-01 23:12:02 +05:30
Sajad Parra
dc46bd263b add options to specify User-Agent in headless template 2022-02-24 12:31:08 +05:30
mzack
54d8f9fdb5 Adding support for payloads in headless protocol 2022-02-04 11:43:42 +01:00
mzack
c701e2ad4a adding support for interactsh 2022-01-31 08:52:36 +01:00
mzack
5d699cdde0 Adding support for full navigation history to headless matchers 2021-12-29 09:48:46 +01:00
Sandeep Singh
f78ff42dbe
Merge branch 'dev' into more-protocols 2021-11-09 18:27:13 +05:30
Ice3man543
8ad3ebcd05 Made code changes as per review comments 2021-11-05 03:01:41 +05:30
Ice3man543
645ae30a47 Moved to an enum for TemplateType in protocols 2021-11-03 19:53:45 +05:30
forgedhallpass
8f6280dc9d refactor: In case of binary data, show a hexadecimal view as well #1080
* small enhancements with regards to dumping responses
2021-11-01 20:45:54 +02:00
forgedhallpass
b91166a034 Merge remote-tracking branch 'origin/dev' into colored_hexdump 2021-11-01 11:57:45 +02:00
Alexey Zhuchkov
463c1c0142 Use template path in output request 2021-10-30 14:28:30 +03:00
forgedhallpass
04e3c0165a feat: In case of binary data, show a hexadecimal view as well #1080 2021-10-30 13:17:47 +03:00
forgedhallpass
435ec5cd5d Do not show AND matcher information in the command line output if debug is not enabled #1081 2021-10-12 20:06:55 +03:00
forgedhallpass
08cd7a4ba7 [feature] Add coloring to debug information #999
* extracted common logic and made sure that all requests implement the same interface
2021-10-01 16:52:38 +03:00
forgedhallpass
898c9431b5 [feature] Add coloring to debug information #999
* corrected/renamed receivers from one character names to human-readable format
2021-10-01 14:30:04 +03:00
forgedhallpass
a00083e221 [feature] Add coloring to debug information #999
* remove some of the code duplication with a different approach
2021-10-01 14:24:45 +03:00
forgedhallpass
aa78b58866 [feature] Add coloring to debug information #999 [WIP]
* if there are multiple matchers, the response is only displayed once, with all the matching values colored
2021-09-30 21:12:44 +03:00
forgedhallpass
4be6b3cc96 [feature] Add coloring to debug information #999 [WIP]
TODO:
* if there are multiple matchers, make sure the response is only displayed once, with all the matching values colored
* remove code duplication from the request.go files
2021-09-29 19:43:46 +03:00
forgedhallpass
448388a825 Added TODOs next to un-used parameters 2021-08-23 16:21:18 +03:00
Mzack9999
6a83e55e40 Progress tracking logic
- Methods refactor
- Counter logic fixes
2021-03-02 02:22:15 +01:00
Ice3man543
8a40fac305 Misc changes to headless 2021-03-01 14:20:56 +05:30
Ice3man
58237f103a
Add headless chrome based templates support (#562) 2021-02-21 16:31:34 +05:30