79 Commits

Author SHA1 Message Date
Tarun Koyalwar
4f93520e47
javascript protocol for scripting (includes 15+ proto libs) (#4109)
* rebase js-layer PR from @ice3man543

* package restructuring

* working

* fix duplicated event & matcher status

* fix lint error

* fix response field

* add new functions

* multiple minor improvements

* fix incorrect stats in js protocol

* sort output metadata in cli

* remove temp files

* remove dead code

* add unit and integration test

* fix lint error

* add jsdoclint using llm

* fix error in test

* add js lint using llm

* generate docs of libs

* llm lint

* remove duplicated docs

* update generated docs

* update prompt in doclint

* update docs

* temp disable version check test

* fix unit test and add retry

* fix panic in it

* update and move jsdocs

* updated jsdocs

* update docs

* update container platform in test

* dir restructure and adding docs

* add api_reference and remove markdown docs

* fix imports

* add javascript design and contribution docs

* add js protocol documentation

* update integration test and docs

* update doc ext mdx->md

* minor update to docs

* new integration test and more

* move go libs and add docs

* gen new net docs and more

* final docs update

* add new devtool

* use fastdialer

* fix build fail

* use fastdialer + network sandbox support

* add reserved keyword 'Port'

* update Port to new syntax

* misc update

* always enable templatectx in js protocol

* move docs to 'js-proto-docs' repo

* remove scrapefuncs binary

---------

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
Ramana Reddy
237cb0bc79 remove deprecated dsl helper func 2023-08-07 23:50:16 +05:30
dependabot[bot]
78f4d9f5ba
chore(deps): bump github.com/projectdiscovery/fastdialer from 0.0.26 to 0.0.28 in /v2 (#3779)
* chore(deps): bump github.com/projectdiscovery/fastdialer in /v2

Bumps [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) from 0.0.26 to 0.0.28.
- [Release notes](https://github.com/projectdiscovery/fastdialer/releases)
- [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.0.26...v0.0.28)

---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/fastdialer
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump retryabledns to 0.28

* Update the retryabledns

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: shubhamrasal <shubhamdharmarasal@gmail.com>
2023-06-08 11:35:53 +05:30
Sandeep Singh
d55ad995d0
tlsx dep update (#3633)
* tlsx dep update

* upgrde httpx => 1.3.0

* Fix check for OS made in MustDisableSandbox() (#3631)

Signed-off-by: iamargus95 <kamathsuraj95@gmail.com>

* Fix wrong template loading in dev branch (#3629)

* Templates wrong loading

* Add tests to cover following scenarios

- check optional fields only if template loaded
- it should return warning only if template is loaded

* enable color in windows (#3634)

* enable color in windows

* fixed win workflow

* typo update

---------

Signed-off-by: iamargus95 <kamathsuraj95@gmail.com>
Co-authored-by: Ramana Reddy <ramanaredy.manda@gmail.com>
Co-authored-by: Suraj Kamath <kamathsuraj95@gmail.com>
Co-authored-by: Shubham Rasal <shubham@projectdiscovery.io>
2023-05-02 16:22:41 +05:30
Mzack9999
34000e1b6b
deps update (#3477)
* deps update

* fixing gologger via callback
2023-03-29 02:42:08 +05:30
Ramana Reddy
7c5fb7fe43
migrate dsl helper functions to dsl repo (#3461)
* migrate dsl pkg code to dsl repo

* fix lint error

* upgrade dsl dependency

* upgrade deps

---------

Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
2023-03-27 18:48:50 +05:30
xm1k3
f26e01551e
resolve() helper function (#3321)
* started the implementation of resolve helper function

* fixes go mod and sum

* fixes and use makeDslWithOptionalArgsFunction signature

* added tests

* added more dnsTypes based on dnsx docs

* used dns client pool

* dsl functions

* fixes on dnsclientpool, added init()

* go mod tidy

* go mod tidy

* dsl signature makeMultiSignatureDslFunction

* error on len

* managed mx

* fix on mx record

* dns types managed with FirstNonZero func

* error handling

* utils to stable version

* version bump

* fixing var name

---------

Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
2023-03-04 04:21:24 +05:30
Mzack9999
62af038617
Add template sign/verify functionality (#3029)
* add template sign/verify functionality

* fixing syntax
2023-02-26 03:54:46 +05:30
xm1k3
cc2f796d2f
Helper function for IP Formats (#3286)
* implemented ip_format helper function

* added tests on ip_format() helper

* fixes on logic

* fixes related to mapcidr docs

* better error value

* fixes + unit test to check index 11

* added call for integration tests

* fixes on dsl-functions number

---------

Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
2023-02-17 01:47:40 +05:30
Ice3man
7e7bb1ed0a
AES CBC PKCS5Padding helper function update (#3287)
* Added DSL helper functions for CVE + misc

* Added aes_cbc with pkcspadding

* Misc

* Misc

* Misc

* Removed debug statement

* Misc

* Misc

* Fixed tests
2023-02-09 20:22:42 +05:30
Dwi Siswanto
94ec553234
Add generate_jwt & json_{minify,prettify} helper functions (#3160)
* feat(dsl): add generate_jwt helper func

* feat(dsl): add json_{minify,prettify} & quote_escape

* update(dsl): change type of data var to map[string]interface{}

* docs(dsl): list valid algos for generate_jwt

* test(dsl): add test case for json_{minify,prettify} & quote_escape

* update(dsl): refactor generate_jwt

* fix(lint): use time.Until instead of t.Sub(time.Now()) (gosimple)

* revert(dsl): remove quote_escape func

* ability to fuzz jwt noNe algorithm

* fix lint error

* jwt dsl minor improvement

Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
2023-01-15 21:58:51 +05:30
Mzack9999
6cdff62381
removing unwanted print debug (#2995) 2022-12-28 21:19:06 +05:30
forgedhallpass
0295ca19bc
Add split DSL function (#2838)
* Add support for showing overloaded DSL method signatures

* Add `split` DSL function #2837

* fixing lint warnings

* replacing faulty regex with strings methods

Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
Co-authored-by: mzack <marco.rivoli.nvh@gmail.com>
2022-11-14 06:08:12 +05:30
Ice3man
1fd83c03bc
Improved dsl test by removing coloring + fix datatime tests (#2654)
* Improved dsl test by removing coloring + fix datatime tests

* Fixed tests for dsl coloring
2022-10-06 03:18:05 +05:30
Myung-jong Kim
9eea441b0e
Add sort(list), sort(string), uniq(list), uniq(string) helper functions (#2372)
* Add feature in join() to sort a single string slice

Signed-off-by: Myung-jong Kim <mjkim610@gmail.com>

* Add sort helper function and related tests

Signed-off-by: Myung-jong Kim <mjkim610@gmail.com>

* Add uniq helper function and related tests

Signed-off-by: mjkim610 <mjkim610@gmail.com>

Signed-off-by: Myung-jong Kim <mjkim610@gmail.com>
Signed-off-by: mjkim610 <mjkim610@gmail.com>
2022-09-27 02:59:13 +05:30
Sami
f3eb0daa39
additional dsl functions (#2550)
* additional dsl functions

* avoid conversion at each iteration
2022-09-08 14:25:34 +05:30
forgedhallpass
fc0763641f
New dsl functions (#2545)
* Update GO version to 1.18

* Removed redundant entry from the .gitignore file

* Added new DSL functions

to_unix_time(input string, optionalLayout string) int64
hex_to_dec(input string) float64
oct_to_dec(input string|number) float64
bin_to_dec(intput string|number) float64

* Notify if debug is enabled when a proxy cannot be validated

* Documentation: Go version requirement updated to 1.18

* test fix: Timezone agnostic date expectation in the assertion

* code review: extracted the default date-time layouts into a global variable
2022-09-07 00:44:29 +05:30
James Turner
42a0732d68
Add sha512 support (#2517) 2022-08-31 12:36:02 +05:30
M. Ángel Jimeno
62a4e0aa52
Return wrapped errors for DSL compilation problems (#2492)
This allows the DSL help information to be printed when in debug mode.

Fixes #2481
2022-08-29 13:41:32 +05:30
51pwn
606c361b2a
Add substr and aes_cbc DSL functions (#2361)
* 1、add DSL substr for #2304 By @hktalent
substr('xxtestxxx',2)。   testxxx
substr('xxtestxxx',2,-2)  testx
substr('xxtestxxx',2,6)   test

2、add DSL aes_cbc for #2243 By @hktalent
aes_cbc("key111key111key111key111", "dataxxxxxxdataxxxxxxdataxxxxxxdataxxxxxxdataxxxxxx")

3、fixed An error occurs when running nuclei with multiple instances #2301 By @hktalent

* refactoring helpers

* removing unwanted mutex

* commenting out test

* removing aes_cbc test due to random iv

Co-authored-by: 51pwn <51pwn@51pwn.com>
Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
2022-08-25 15:50:08 +05:30
Ice3man
8f313629b8
Memory usage optimizations (#2350)
* Replaced strings.Replaced with fasttemplate reducing allocations

Custom template parsing logic was replaced with fasttemplate package for reducing
allocations in the replacer.Replace hotpath leading to allocation reduction which
accounted for 30% of total nuclei allocations.

$ go test -bench=. -benchmem
goos: darwin
goarch: arm64
pkg: github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/replacer
BenchmarkReplacer-8               837232              1422 ns/op            2112 B/op         31 allocs/op
BenchmarkReplacerNew-8           3672765               320.3 ns/op            48 B/op          4 allocs/op

* Fixed tests failing

* Use pre-compiled map of DSL expressions

* Reworked expression parsing logic to reduce memory allocations

$ go test -bench=. -benchmem
goos: darwin
goarch: arm64
pkg: github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/expressions
BenchmarkEvaluate-8        31560             37769 ns/op           31731 B/op        265 allocs/op
BenchmarkEvaluateNew-8       109144              9621 ns/op            6253 B/op        116 allocs/op
2022-08-23 13:16:41 +05:30
Sajad
c4ba2b4edb
Additional helpers (#2359)
* add starts_with, ends_with helper functions

* add test cases, update dsl signatures

* change split_starts_with to line_starts_with
2022-08-06 23:16:03 +05:30
Ice3man
9073b753ca
Added aes_gcm implementation for DSL function (#2196)
* Added aes_gcm implementation for DSL function

* Added integration test for dsl-functions.yaml
2022-06-23 16:16:24 +05:30
forgedhallpass
ef20e0711b refactor: replace date, time, time_format, time_to_string DSL functions to date_time 2022-06-08 20:33:55 +03:00
forgedhallpass
a10d58c6d2 refactor: rename concat_ws DSL function to join 2022-06-08 16:31:33 +03:00
forgedhallpass
145bdaabe5 refactor: extract duplicated hashing logic 2022-06-08 16:11:15 +03:00
forgedhallpass
6d8908c352 refactor: replace hmac_sha1 and hmac_sha256 DSL functions with hmac 2022-06-08 16:00:14 +03:00
James Turner
9d37bd6c0c Add two new DSL helper functions
hmac_sha1 and concat_ws (with seperator) this are helpful in
signing API requests.
2022-06-07 18:26:22 +03:00
forgedhallpass
f3514e9b92 Merge branch 'dev' into new_dsl_functions 2022-06-07 17:26:10 +03:00
M4rtin Hsu
aebd32b198
Add decimal to hexadecimal helper function (#2076)
* Add decimal to hexadecimal auxiliary functions

* Fixed unit test

* Modify the helper function name and check the unit test.

* dsl function update

Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
2022-06-03 02:17:35 +05:30
forgedhallpass
096f34e4e3 refactor: DSL date/time functions 2022-05-24 13:38:26 +03:00
forgedhallpass
346db4cf15 refactor: timetostring DSL to time_to_string 2022-05-24 13:11:55 +03:00
Rahmat
feae805834 Add hmac_sha256 dslExpression 2022-05-24 12:59:09 +03:00
Rahmat
f2ca75c536 Add time_format dslExpression 2022-05-24 12:47:56 +03:00
Ice3man
ea4e8cc265
Added version comparison helpers (#1783)
* Added version comparison helpers

* Added multiple condition example

* Added variadic version of compare_versions

* Fixed failing DSL integration test
2022-04-07 00:48:34 +05:30
Ice3man
159d84465d Fixed failing DSL tests 2022-03-29 17:51:40 +05:30
Xavier Stevens
4a65097194 Squashed commit of the following:
commit b590de2de14923e4cb35dd19845e12833a4ccbfa
Author: Xavier Stevens <xstevens@users.noreply.github.com>
Date:   Wed Mar 23 08:29:37 2022 -0700

    Updated rand_ip to use variadic args

commit ea883be8c0aa56174a1301252129289334659f3f
Author: Xavier Stevens <xstevens@users.noreply.github.com>
Date:   Tue Mar 22 16:59:19 2022 -0700

    Updated rand_ip function to handle multiple CIDRs
2022-03-23 08:37:05 -07:00
mzack
6213454cb3 small code refactor 2022-03-22 15:16:22 +01:00
skhalsa-sigsci
63eccc85ba random ip helper function 2022-03-21 17:39:10 -07:00
Ice3man
be0e4b227c Pad date and time helper strings with double digits 2022-03-21 16:32:21 +05:30
Ice3man
bebdb29706 Added time + zlib dsl functions + misc 2022-03-16 14:12:26 +05:30
mzack
f7c2226c39 Adding number/string conversion helpers 2022-03-10 10:57:59 +01:00
Ice3man
ca7e30b3fb Added tests for DSL function + misc 2022-03-10 11:56:11 +05:30
Ice3man
deca1095ed Added gzip_decode helper function 2022-02-25 22:37:50 +05:30
forgedhallpass
7f6c72853f feat: introduce 'concat' DSL function #1502 2022-01-17 13:32:15 +02:00
mzack
256c1f411e fixing mmh3 signed int32 calculation 2022-01-04 18:10:20 +01:00
mzack
357916a6e2 Replacing murmur3 with internal safe-pointer version 2022-01-02 09:09:10 +01:00
forgedhallpass
85e0b96d51
bug: fixed couple of bugs in the DSL functions (#1372)
* feat: Improve DSL function UX #1295

Sort the output signatures

* feat: Improve DSL function UX #1295

Sort the output signatures.
Lint: simplified the sorting.

* bug: fixed couple of bugs in the DSL functions

Input number parameters are stored as float64 types, hence the type conversion should happen accordingly. Affected functions:
* rand_int
* wait_for
* unix_time
* rand_text_numeric

Added tests for all functions.
Related: #1261

* bug: fixed couple of bugs in the DSL functions

Handle cases when the optional input character set is an empty string. Affected methods:
* rand_char
* rand_base

* bug: fixed couple of bugs in the DSL functions

Change rand_char to return a one character string, instead of the character code

* refactor: Minor integration test changes to show the actual and expected result numbers

* test: Added integration test for all existing DSL functions

* test: Added integration test for all existing DSL functions

Fixing linter issues.

* feat: Add "repeat" DSL function

* test: Add "repeat" DSL function
2021-12-15 19:33:57 +05:30
forgedhallpass
10cf1053f8
feat: Improve DSL function UX #1295 (#1351)
* feat: Improve DSL function UX #1295

Sort the output signatures

* feat: Improve DSL function UX #1295

Sort the output signatures.
Lint: simplified the sorting.
2021-12-09 14:02:01 +05:30
forgedhallpass
79aed22d46 feat: Improve DSL function UX #1295
Added support for letting people know if:
* the DSL expression does not return a boolean value
* an invalid custom function signature was provided and then display all available function signatures
* an invalid function was provided and then display the correct signature

Unified the DSL function names to use snake case. The old signatures are also kept for backward compatibility.
2021-12-07 17:34:36 +02:00