43 Commits

Author SHA1 Message Date
Tarun Koyalwar
c35162c8ef
nuclei v3 bug fixes (#4176)
* store and generate signer keys

* fix trailing newline in code_response

* fix formatting and update error string

* fix integration test

* fix rsaSigned code integration test

* bug fixes , docs and more

* bump go -> 1.21

* use 'response' as default part in code templates

* disable sourcemaps for all js runtimes

* disable eval function

* rewrite file validation in sandbox mode

* sandbox file read improvements + minor refactor

* refactor sign and verify logic

* fix panic and missing id in code protocol

* disable re-signing code protocol templates

* fix code resigning in tests

* allow -lfa in test for signing templates

* start index from 1 in flow and multiproto

* remove testfiles

* add python in integration test

* update code protocol docs

* add python engine in template

* rework template signer

* fix integration test and more

* reworked template signer

* fix lint error

* display signature stats

* update docs

* add user fragment to signature

* use md5 to generate fragment

* update docs with code re-sign

* misc updates

* public crt update

* remove workflow info statement

* fix printing issues

* refactor preprocessor logic

* remove debug statement

* fix failing example test

* go mod tidy

---------

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
2023-10-13 13:17:27 +05:30
Tarun Koyalwar
4f93520e47
javascript protocol for scripting (includes 15+ proto libs) (#4109)
* rebase js-layer PR from @ice3man543

* package restructuring

* working

* fix duplicated event & matcher status

* fix lint error

* fix response field

* add new functions

* multiple minor improvements

* fix incorrect stats in js protocol

* sort output metadata in cli

* remove temp files

* remove dead code

* add unit and integration test

* fix lint error

* add jsdoclint using llm

* fix error in test

* add js lint using llm

* generate docs of libs

* llm lint

* remove duplicated docs

* update generated docs

* update prompt in doclint

* update docs

* temp disable version check test

* fix unit test and add retry

* fix panic in it

* update and move jsdocs

* updated jsdocs

* update docs

* update container platform in test

* dir restructure and adding docs

* add api_reference and remove markdown docs

* fix imports

* add javascript design and contribution docs

* add js protocol documentation

* update integration test and docs

* update doc ext mdx->md

* minor update to docs

* new integration test and more

* move go libs and add docs

* gen new net docs and more

* final docs update

* add new devtool

* use fastdialer

* fix build fail

* use fastdialer + network sandbox support

* add reserved keyword 'Port'

* update Port to new syntax

* misc update

* always enable templatectx in js protocol

* move docs to 'js-proto-docs' repo

* remove scrapefuncs binary

---------

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
Tarun Koyalwar
f7fe99f806
add flow support in template (i.e javascript scripting) (#4015)
* add flow logic

* progress

* working POC

* fix string slice normalization issue in variables

* update

* fix nil panic

* remove poll()

* load file with sandbox and more

* fix failing integration tests

* JS: log: print in vardump format

* fix missing id in protocols

* fix proto prefix in template context

* flow: add unit tests

* conditional flow support using flow

* fix proto callbacks + more unit tests

* adds integration test

* conditional flow: check if req has any matchers

* fix lint error

* deprecate iterate-all+ missing multi-proto implementation

* fix ip input in raw request

* JS: feat dedupe object+ more builtin funcs

* feat: hide protocol result using hide

* feat: async execution

* complete async execution support

* fix condition-flow without any matchers

* refactor: template executer package (tmplexec)

* flow executor working

* fix data race in templateCtx

* templateCtx redesign

* fix failing unit test

* add multiprotocol support to deprecated syntax

* fix race condition in utils & tlsx

* add documentation in flow package

* remove regions.txt file

* fix minor issue with self contained templates

* fix typos of copilot

* dep + misc update

* fix reqID: use req.Type instead of template.Type

---------

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-08-31 18:03:01 +05:30
Tarun Koyalwar
8125b6805c resolve merge conflicts with dev 2023-08-04 20:21:22 +05:30
Dogan Can Bakir
163bc22281
add headless options flag (#3951)
* add headless options flag

* disable some tests for windows

* disable interactsh tests on darwin

* disable network/hex.yaml on windows

* make DisableOn func
2023-07-28 21:20:57 +05:30
Tarun Koyalwar
6bdef68734
ignore version parsing error (#3984)
* ignore version parsing error

* hide no parameter error

* integration test+ DEBUG.md

* typo fix in DEBUG.md

* go mod tidy

---------

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-07-28 21:04:02 +05:30
Tarun Koyalwar
bdf77005d6 resolve merge conflicts 2023-06-27 20:21:14 +05:30
Mzack9999
fa199ed3b3
Improving clientpool with client certificates (#3851)
* Improving clientpool with client certificates

* adding test case

* Revert "Merge branch 'dev' into issue-3800-client-cert"

This reverts commit 7f057d742f4b9bda8e83b2052e29617b86b6776d, reversing
changes made to 7297cebcf8bb0f88961b644fc2ac7c040df8ffd9.

* Revert "Revert "Merge branch 'dev' into issue-3800-client-cert""

This reverts commit 2053a248a0cdc2002e0b4b4faa3472cf11c29760.

* go fmt

---------

Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-06-26 17:45:12 +05:30
Mzack9999
a7fb15d0bd
Adding support for code templates (#2930)
* Adding support for code templates

* adding support for python, powershell and echo (test)

* removing debug code

* introducing command + trivial trust store mechanism

* updating tests

* adding basic tests

* removing deprecated oracle

* mod tidy

* adding signature proto with debug prints

* removing debug code

* fixing test

* fixing param order

* improving test conditional build

* disable file+offlinehttp+code with cloud

* adding env vars

* removing debug code

* reorganizing test folders

* adding code template test prototype with dummy priv/pub keys

* bump go to 1.20

* fixing go version

* fixing lint errors

* adding fatal on pub-key test failure

* switching to ecdsa asn1

* removing unused signature

* fixing signature

* adding more tests

* extending core with engine args + powershell win test

* adding unsigned code test

* skip template signing in particular test case

* improving test coverage

* refactoring key names + adding already signed algo

* removing debug code

* fixing syntax

* fixing lint issues

* removing test template

* fixing dns tests path

* output fmt

* adding interact

* fixing lint issues

* adding -sign cli helper

* fixing nil pointer + parse inline keys

* making rsa default

* adding code prot. ref

* moving file to correct loc

* moving test

* Issue 3339 headless fuzz (#3790)

* Basic headless fuzzing

* Remove debug statements

* Add integration tests

* Update template

* Fix recognize payload value in matcher

* Update tempalte

* use req.SetURL()

---------

Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>

* Auto Generate Syntax Docs + JSONSchema [Fri Jun  9 00:23:32 UTC 2023] 🤖

* Add headless header and status matchers (#3794)

* add headless header and status matchers

* rename headers as header

* add integration test for header+status

* fix typo

* add retry to py-interactsh integration test

---------

Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
Co-authored-by: Shubham Rasal <shubham@projectdiscovery.io>
Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Dogan Can Bakir <65292895+dogancanbakir@users.noreply.github.com>
Co-authored-by: Tarun Koyalwar <45962551+tarunKoyalwar@users.noreply.github.com>
2023-06-09 20:54:24 +05:30
Tarun Koyalwar
e1d3f474a4
support for dynamic variables in template context (multi protocol execution) (#3672)
* multi proto request genesis

* adds template context dynamic vars

* feat: proto level resp variables

* remove proto prefix hacky logic

* implement template ctx args

* remove old var name logic

* improve AddTemplateVars func

* add multi proto comments+docs

* vardump with sorted keys

* fix race condition in ctx args

* default initialize ctx args

* use generic map

* index variables with multiple values

* fix nil cookies

* use synclock map

* fix build failure

* fix lint error

* resolve merge conflicts

* multi proto: add unit+ integration tests

* fix unit tests

* Issue 3339 headless fuzz (#3790)

* Basic headless fuzzing

* Remove debug statements

* Add integration tests

* Update template

* Fix recognize payload value in matcher

* Update tempalte

* use req.SetURL()

---------

Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>

* Auto Generate Syntax Docs + JSONSchema [Fri Jun  9 00:23:32 UTC 2023] 🤖

* Add headless header and status matchers (#3794)

* add headless header and status matchers

* rename headers as header

* add integration test for header+status

* fix typo

---------

Co-authored-by: Shubham Rasal <shubham@projectdiscovery.io>
Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Dogan Can Bakir <65292895+dogancanbakir@users.noreply.github.com>
2023-06-09 19:52:56 +05:30
Mzack9999
6f4b1ae48a
Replacing ccache with generic gcache (#3523)
* Replacing ccache with generic gcache

* fixing lint issues

* removing unecessary hashing + using errorutils

* making test more tolerant

* removing dead code + refactor

* removing redundant code

* removing race

* maint

* moving code

* adding more iterations

* note + typo

* temporary fixing stop-at-first-match with interact

* wrapping internal map with mux

* sort before running integration test

* fix deadlock in requestShouldStopAtFirstMatch

* add timeout to integration_test workflow

* attempting to remove outer lock

* adds interactsh protocol tests in integration_test

---------

Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
2023-04-16 23:19:35 +05:30
Tarun Koyalwar
a185348194
fix integration test (#3506)
* fix integration test

* fix interactsh fatal error
2023-04-04 10:09:52 +08:00
Mzack9999
1b7585476e simplify execution logic 2023-03-17 17:31:28 +01:00
Mzack9999
6a0db1c234 restoring tests 2023-03-17 16:56:14 +01:00
Mzack9999
c115768b8e investigating tests 2023-03-17 15:39:12 +01:00
Shubham Rasal
572c8eb780
Issue 2987 fuzz options (#3355)
* Add override fuzzing type and mode flags

* Update english readme

* Fix failing tests

* Add the integration tests

- validate the command line overriding type and mode for fuzzing
2023-03-06 16:56:38 +05:30
Tarun Koyalwar
628628893c
fix make http request inconsistencies (#3243)
* fix make http request inconsistencies

* remove parameters from http vars

* fix trailingslash unit test

* naming conventions: best practices

* fix publish docs action

* remove branch ref from push
2023-02-01 17:23:28 +05:30
Tarun Koyalwar
0b2a3e296a
fix url encoding issues and inconsistencies (#3211)
* fix url encoding issues

* complete requested changes and improvements

* fix missing issue-tracker-config.yaml

* fuzz: deepcopy and use urlutil.Params
2023-01-24 22:04:52 +05:30
LuitelSamikshya
909aa4cd79 updates to execute func in integration-test.go 2022-10-13 11:05:10 -05:00
Sami
0aac36a44b
added custom config flag (#2399)
* added custom config flag

* config.yaml file in custom  directory

* lint error fix

* few updates and error checks

* fix lint error

* copy config.yaml file if the dest folder does not exist

* lint error check

* added integration test

* improved test cases

* lint error fix
2022-08-25 16:10:07 +05:30
Mzack9999
1c332bb85b
Improving RFC request/response passive parsing (#2192)
* Improving RFC request/response passive parsing

* adding test
2022-07-11 22:43:10 +05:30
Mzack9999
dd3b0a3cfc
Increasing buffer size on AND condition (#2030)
* Increasing buffer size on AND condition

* adding integration tests

* fixing tests data path
2022-05-30 13:45:28 +05:30
Ice3man
a031ef32b4 Fixed nuclei go example + added relevant integration test 2022-04-28 01:02:06 +05:30
LuitelSamikshya
0ca16c4248 integration test for temp update/install 2022-03-21 16:11:58 -05:00
LuitelSamikshya
9128e04d07 integration tests for template path 2022-03-08 09:50:08 -06:00
mzack
19d2b80a1c adding ssl test cases + tcpserver refactor 2022-01-27 21:29:28 +01:00
Sajad
5200bcd94f
Whois Protocol Support (using rdap) (#1354)
* init rdap

* add an integration test, option to supply RDAP server to execute the request on

* add rdap protocolMappings

* add debug info, add IP, ASN query type support

* rename rdap to whois, Host to Query in template

* rename pending rdap to whois

* remove port from whois varaiables

* set Host variable even if input is not a parsable url
2021-12-16 17:08:02 +05:30
forgedhallpass
85e0b96d51
bug: fixed couple of bugs in the DSL functions (#1372)
* feat: Improve DSL function UX #1295

Sort the output signatures

* feat: Improve DSL function UX #1295

Sort the output signatures.
Lint: simplified the sorting.

* bug: fixed couple of bugs in the DSL functions

Input number parameters are stored as float64 types, hence the type conversion should happen accordingly. Affected functions:
* rand_int
* wait_for
* unix_time
* rand_text_numeric

Added tests for all functions.
Related: #1261

* bug: fixed couple of bugs in the DSL functions

Handle cases when the optional input character set is an empty string. Affected methods:
* rand_char
* rand_base

* bug: fixed couple of bugs in the DSL functions

Change rand_char to return a one character string, instead of the character code

* refactor: Minor integration test changes to show the actual and expected result numbers

* test: Added integration test for all existing DSL functions

* test: Added integration test for all existing DSL functions

Fixing linter issues.

* feat: Add "repeat" DSL function

* test: Add "repeat" DSL function
2021-12-15 19:33:57 +05:30
forgedhallpass
974cbfb35e
feat: Re-run failed integration tests in debug mode (#1367)
* If the tests are executed through GitHub actions and there are failed integration tests, they will be re-executed with verbose output to help figuring out the underlying issues.
* Added some grouping to make reading the logs easier

Ticket: #1365
2021-12-14 18:13:53 +02:00
Mzack9999
8336421179
Improving headless test cases (#1313)
* Adding headless test cases
2021-12-04 23:10:11 +05:30
forgedhallpass
47340f06b0 refactor: uniformly sorted imports 2021-11-25 17:09:20 +02:00
Ice3man543
c1a35b3ff9 Merge branch 'dev' of https://github.com/projectdiscovery/nuclei into more-protocols 2021-11-11 17:21:25 +05:30
Ice3man543
9b364080ea Fixed some edge cases with status codes in redirect chains 2021-11-09 17:55:42 +05:30
Ice3man543
8ad3ebcd05 Made code changes as per review comments 2021-11-05 03:01:41 +05:30
Ice3man543
390ca8b3c6 Merge from dev 2021-11-03 18:58:00 +05:30
Ice3man543
bb05be7b95 Added integration tests for websocket + misc fixes 2021-11-01 15:47:20 +05:30
Jop Zitman
81102750a1 Implement integration tests for remote template and workflow urls. 2021-10-14 23:33:08 +02:00
forgedhallpass
40d88d2304 In-lined error checks, reduced scope of error variables, introduced new error variables instead of re-using them 2021-08-31 12:55:52 +03:00
Ice3man543
8d0b5d6203 RES-120 Added functional testing implementation 2021-07-30 15:29:12 +05:30
Ice3man543
13c67a62bd Fixed template-update + added workflow tests 2021-02-27 20:54:22 +05:30
Ice3man543
f0e0c1e7d0 Misc changes to integration test 2021-02-27 12:42:35 +05:30
Ice3man543
2a0dcc1ee3 More tests for dns and network + misc 2021-02-27 02:23:06 +05:30
Ice3man543
b33bc83b0b Start of work on testing infrastructure 2021-02-25 23:32:43 +05:30