111 Commits

Author SHA1 Message Date
Tarun Koyalwar
ab0f0ea92b
Printing bug fixes (#4238)
* fix printing issue of unsigned workflows

* add NUCLEI_LOG_ALL env variable

* enable tlsx -dns and update print msg

* misc update

---------

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-10-16 14:34:52 +05:30
Tarun Koyalwar
c35162c8ef
nuclei v3 bug fixes (#4176)
* store and generate signer keys

* fix trailing newline in code_response

* fix formatting and update error string

* fix integration test

* fix rsaSigned code integration test

* bug fixes , docs and more

* bump go -> 1.21

* use 'response' as default part in code templates

* disable sourcemaps for all js runtimes

* disable eval function

* rewrite file validation in sandbox mode

* sandbox file read improvements + minor refactor

* refactor sign and verify logic

* fix panic and missing id in code protocol

* disable re-signing code protocol templates

* fix code resigning in tests

* allow -lfa in test for signing templates

* start index from 1 in flow and multiproto

* remove testfiles

* add python in integration test

* update code protocol docs

* add python engine in template

* rework template signer

* fix integration test and more

* reworked template signer

* fix lint error

* display signature stats

* update docs

* add user fragment to signature

* use md5 to generate fragment

* update docs with code re-sign

* misc updates

* public crt update

* remove workflow info statement

* fix printing issues

* refactor preprocessor logic

* remove debug statement

* fix failing example test

* go mod tidy

---------

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
2023-10-13 13:17:27 +05:30
Tarun Koyalwar
4f93520e47
javascript protocol for scripting (includes 15+ proto libs) (#4109)
* rebase js-layer PR from @ice3man543

* package restructuring

* working

* fix duplicated event & matcher status

* fix lint error

* fix response field

* add new functions

* multiple minor improvements

* fix incorrect stats in js protocol

* sort output metadata in cli

* remove temp files

* remove dead code

* add unit and integration test

* fix lint error

* add jsdoclint using llm

* fix error in test

* add js lint using llm

* generate docs of libs

* llm lint

* remove duplicated docs

* update generated docs

* update prompt in doclint

* update docs

* temp disable version check test

* fix unit test and add retry

* fix panic in it

* update and move jsdocs

* updated jsdocs

* update docs

* update container platform in test

* dir restructure and adding docs

* add api_reference and remove markdown docs

* fix imports

* add javascript design and contribution docs

* add js protocol documentation

* update integration test and docs

* update doc ext mdx->md

* minor update to docs

* new integration test and more

* move go libs and add docs

* gen new net docs and more

* final docs update

* add new devtool

* use fastdialer

* fix build fail

* use fastdialer + network sandbox support

* add reserved keyword 'Port'

* update Port to new syntax

* misc update

* always enable templatectx in js protocol

* move docs to 'js-proto-docs' repo

* remove scrapefuncs binary

---------

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
Tarun Koyalwar
f7fe99f806
add flow support in template (i.e javascript scripting) (#4015)
* add flow logic

* progress

* working POC

* fix string slice normalization issue in variables

* update

* fix nil panic

* remove poll()

* load file with sandbox and more

* fix failing integration tests

* JS: log: print in vardump format

* fix missing id in protocols

* fix proto prefix in template context

* flow: add unit tests

* conditional flow support using flow

* fix proto callbacks + more unit tests

* adds integration test

* conditional flow: check if req has any matchers

* fix lint error

* deprecate iterate-all+ missing multi-proto implementation

* fix ip input in raw request

* JS: feat dedupe object+ more builtin funcs

* feat: hide protocol result using hide

* feat: async execution

* complete async execution support

* fix condition-flow without any matchers

* refactor: template executer package (tmplexec)

* flow executor working

* fix data race in templateCtx

* templateCtx redesign

* fix failing unit test

* add multiprotocol support to deprecated syntax

* fix race condition in utils & tlsx

* add documentation in flow package

* remove regions.txt file

* fix minor issue with self contained templates

* fix typos of copilot

* dep + misc update

* fix reqID: use req.Type instead of template.Type

---------

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-08-31 18:03:01 +05:30
Tarun Koyalwar
8125b6805c resolve merge conflicts with dev 2023-08-04 20:21:22 +05:30
Mzack9999
305ac6a143
Bugfix 3871 workflow concurrency (#3903)
* recursion waitgroup fix

* cleanup
2023-07-06 17:38:24 +05:30
Mzack9999
a7fb15d0bd
Adding support for code templates (#2930)
* Adding support for code templates

* adding support for python, powershell and echo (test)

* removing debug code

* introducing command + trivial trust store mechanism

* updating tests

* adding basic tests

* removing deprecated oracle

* mod tidy

* adding signature proto with debug prints

* removing debug code

* fixing test

* fixing param order

* improving test conditional build

* disable file+offlinehttp+code with cloud

* adding env vars

* removing debug code

* reorganizing test folders

* adding code template test prototype with dummy priv/pub keys

* bump go to 1.20

* fixing go version

* fixing lint errors

* adding fatal on pub-key test failure

* switching to ecdsa asn1

* removing unused signature

* fixing signature

* adding more tests

* extending core with engine args + powershell win test

* adding unsigned code test

* skip template signing in particular test case

* improving test coverage

* refactoring key names + adding already signed algo

* removing debug code

* fixing syntax

* fixing lint issues

* removing test template

* fixing dns tests path

* output fmt

* adding interact

* fixing lint issues

* adding -sign cli helper

* fixing nil pointer + parse inline keys

* making rsa default

* adding code prot. ref

* moving file to correct loc

* moving test

* Issue 3339 headless fuzz (#3790)

* Basic headless fuzzing

* Remove debug statements

* Add integration tests

* Update template

* Fix recognize payload value in matcher

* Update tempalte

* use req.SetURL()

---------

Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>

* Auto Generate Syntax Docs + JSONSchema [Fri Jun  9 00:23:32 UTC 2023] 🤖

* Add headless header and status matchers (#3794)

* add headless header and status matchers

* rename headers as header

* add integration test for header+status

* fix typo

* add retry to py-interactsh integration test

---------

Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
Co-authored-by: Shubham Rasal <shubham@projectdiscovery.io>
Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Dogan Can Bakir <65292895+dogancanbakir@users.noreply.github.com>
Co-authored-by: Tarun Koyalwar <45962551+tarunKoyalwar@users.noreply.github.com>
2023-06-09 20:54:24 +05:30
Tarun Koyalwar
e1d3f474a4
support for dynamic variables in template context (multi protocol execution) (#3672)
* multi proto request genesis

* adds template context dynamic vars

* feat: proto level resp variables

* remove proto prefix hacky logic

* implement template ctx args

* remove old var name logic

* improve AddTemplateVars func

* add multi proto comments+docs

* vardump with sorted keys

* fix race condition in ctx args

* default initialize ctx args

* use generic map

* index variables with multiple values

* fix nil cookies

* use synclock map

* fix build failure

* fix lint error

* resolve merge conflicts

* multi proto: add unit+ integration tests

* fix unit tests

* Issue 3339 headless fuzz (#3790)

* Basic headless fuzzing

* Remove debug statements

* Add integration tests

* Update template

* Fix recognize payload value in matcher

* Update tempalte

* use req.SetURL()

---------

Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>

* Auto Generate Syntax Docs + JSONSchema [Fri Jun  9 00:23:32 UTC 2023] 🤖

* Add headless header and status matchers (#3794)

* add headless header and status matchers

* rename headers as header

* add integration test for header+status

* fix typo

---------

Co-authored-by: Shubham Rasal <shubham@projectdiscovery.io>
Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Dogan Can Bakir <65292895+dogancanbakir@users.noreply.github.com>
2023-06-09 19:52:56 +05:30
Keith Chason
4d6080f3bc
"Executer" to "Executor" (#3760)
* Fix spelling of "executer" to "executor"

* minor change: use defer file.Close()

---------

Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
2023-06-01 02:28:10 +05:30
Mzack9999
0d2d510689
Adding support for constants (#3692)
* adding support for constants

* fixing typo

* adding integration test

* fixing lint issues

* fixing template syntax
2023-05-25 22:02:35 +05:30
Ice3man
ecc1964ad9 Added optional doNotCache to protocols.ExecuterOptions 2023-02-13 18:00:25 +05:30
Mzack9999
454a883241
Using retryablehttp for templates loading (#3291)
* Using retryablehttp for templates loading

* Update compile_test.go
2023-02-09 19:15:44 +05:30
xm1k3
1e5358b1fa
Improve passive templates error handling (#3098)
* fixes on passive templates

* Auto Generate Syntax Docs + JSONSchema [Thu Dec 29 08:47:22 UTC 2022] 🤖

* removed empty line

* warning management

When passive flag is provided we ignore all templates which are not compatible, without posting misleading errors

* removing redundant code

* skip offline errors with err var

* remove check on debug flag + used errors.Is() to check errors

important note for future refactoring: use errorsutil.Is() instead of errors.Is()

---------

Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
2023-02-07 05:10:38 +05:30
shubhamrasal
b03a90df50 fix template path bug, pass the filepath in executeroptions 2023-01-03 12:17:56 +05:30
shubhamrasal
406d0c1873 Rename function from ParseFromReader to ParseTemplateFromReader 2022-12-22 11:04:16 +05:30
shubhamrasal
cff0d88742 Add tests for parse function
- update the ioutils to io.ReadAll
- remove extra line
2022-12-19 18:51:20 +05:30
shubhamrasal
de3dda0030 Update the Parse function to use ParseFromReader 2022-12-15 17:41:07 +05:30
shubhamrasal
198d84ec25 Add ParseFromReader function 2022-12-13 12:57:47 +05:30
vrenzolaverace
2aaf2a2158
Use utils helpers libraries (#2809) (#2810)
* Use utils helpers libraries (#2809)

* Use utils helpers libraries (#2809)
2022-11-07 01:54:23 +05:30
Ice3man
7e209dad01
Added multi-protocol requests support for templates (#2611) 2022-09-16 23:10:39 +05:30
Ice3man
67d5769cd9
Added initial catalog interface implementation (#2318)
* Added initial catalog interface implementation

* Added OpenFile to Catalog + disk catalog implementation

* Fixed merge issues

Co-authored-by: sandeep <sandeep@projectdiscovery.io>
2022-08-10 11:05:58 -07:00
Mzack9999
3c945f6ae9
Adding stricter check on offline templates list (#2213) 2022-07-11 22:38:07 +05:30
Ice3man
a0ece302d1 Moved variables to template level + misc 2022-03-31 00:54:35 +05:30
Sajad Parra
c6de2ca406 rename few vars, extract duplicate functions to utils 2022-01-24 16:48:12 +05:30
Sajad Parra
d1303797c0 * Add support to run remote template
* Add remote-template-domain config only flag to specify allowed domain list to load remote templates from
2022-01-12 18:33:17 +05:30
Sandeep Singh
2dc877a99f
Merge pull request #1247 from projectdiscovery/research-aws-signing
Adding support for aws request signing
2021-12-20 01:06:13 +05:30
Sajad
5200bcd94f
Whois Protocol Support (using rdap) (#1354)
* init rdap

* add an integration test, option to supply RDAP server to execute the request on

* add rdap protocolMappings

* add debug info, add IP, ASN query type support

* rename rdap to whois, Host to Query in template

* rename pending rdap to whois

* remove port from whois varaiables

* set Host variable even if input is not a parsable url
2021-12-16 17:08:02 +05:30
mzack
c47196bcc8 Merge branch 'dev' into research-aws-signing 2021-12-06 08:17:03 +01:00
Sajad Parra
f60e93aae1 add template level stop at first match option 2021-11-29 16:01:06 +05:30
mzack
38ff8f75b1 adding signature syntax validation 2021-11-17 01:28:35 +01:00
mzack
eb61c519c3 converting to post-processor prototype 2021-11-13 03:17:05 +01:00
mzack
34889d50f8 correcting variable name 2021-11-12 19:58:12 +01:00
mzack
882a156b63 Merge branch 'dev' into research-aws-signing 2021-11-12 19:55:35 +01:00
mzack
041361c45e Adding aws sign prototype 2021-11-12 19:29:45 +01:00
Ice3man543
a10b79d7a8 merge from engine-refactor and dev 2021-10-29 18:26:06 +05:30
Ice3man543
97645dde52 Added new workpool package + Misc refactor 2021-10-27 15:53:04 +05:30
Ice3man543
2a84b9eb44 misc 2021-10-26 20:40:02 +05:30
Ice3man543
a0318ffc8f Started refactor of template compilation + protocol building 2021-10-26 20:36:44 +05:30
Ice3man
f25c7c3a1d
Merge branch 'dev' into more-protocols 2021-10-20 21:29:28 +05:30
Ice3man543
612ab61df4 Reworked self-contained requests to template 2021-10-19 21:29:18 +05:30
Ice3man543
396f17484e Added websocket protocol support to nuclei 2021-09-27 18:02:49 +05:30
Ice3man543
f6e9acf06f More work on other protocol additions 2021-09-23 18:30:44 +05:30
Ice3man543
5c31b75ac7 Added ssl protocol to nuclei 2021-09-22 22:41:07 +05:30
forgedhallpass
0cabce518f Merge branch 'dev' into code_smells 2021-09-02 12:30:43 +03:00
Ice3man543
c3e32f5e15 Added new stats counters + validation fixes 2021-08-31 19:27:26 +05:30
forgedhallpass
40d88d2304 In-lined error checks, reduced scope of error variables, introduced new error variables instead of re-using them 2021-08-31 12:55:52 +03:00
Ice3man543
1946d2ec52 Fixed a crash in workflow loader 2021-08-30 16:58:11 +05:30
Ice3man543
78554252ed Misc changes to workflow parts 2021-08-28 00:27:37 +05:30
Ice3man543
30f6498fe2 Replace error constant with an error type 2021-08-28 00:21:07 +05:30
Ice3man543
08fdc3bdcc Added cache package + misc changes to loaders 2021-08-28 00:15:28 +05:30