* fix: remove undefined errorutil.ShowStackTrace
* feat: add make lint support and integrate with test
* refactor: migrate errorutil to errkit across codebase
- Replace deprecated errorutil with modern errkit
- Convert error declarations from var to func for better compatibility
- Fix all SA1019 deprecation warnings
- Maintain error chain support and stack traces
* fix: improve DNS test reliability using Google DNS
- Configure test to use Google DNS (8.8.8.8) for stability
- Fix nil pointer issue in DNS client initialization
- Keep production defaults unchanged
* fixing logic
* removing unwanted branches in makefile
---------
Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
* bugfix: fix memory blowup using previousEvent for multi-proto execution
* refactor(tmplexec): uses supported protocol types
Signed-off-by: Dwi Siswanto <git@dw1.io>
* add co-author
Co-authored-by: Nakul Bharti <knakul853@users.noreply.github.com>
Signed-off-by: Dwi Siswanto <git@dw1.io>
* refactor(tmplexec): mv builder inside loop scope
Signed-off-by: Dwi Siswanto <git@dw1.io>
* refactor(tmplexec): skip existing keys in `FillPreviousEvent`
The `FillPreviousEvent` func was modified to
prevent overwriting/duplicating entries in the
previous map.
It now checks if a key `k` from
`event.InternalEvent` already exists in the
previous map. If it does, the key is skipped. This
ensures that if `k` was already set (potentially
w/o a prefix), it's not re-added with an `ID_`
prefix.
Additionally, keys in `event.InternalEvent` that
already start with the current `ID_` prefix are
also skipped to avoid redundant prefixing.
This change simplifies the logic by removing the
`reqTypeWithIndexRegex` and directly addresses the
potential for duplicate / incorrectly prefixed
keys when `event.InternalEvent` grows during
protocol request execution.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* chore(tmplexec): naming convention, `ID` => `protoID`
Signed-off-by: Dwi Siswanto <git@dw1.io>
* chore(tmplexec): it's request ID lol sorry
Signed-off-by: Dwi Siswanto <git@dw1.io>
---------
Signed-off-by: Dwi Siswanto <git@dw1.io>
Co-authored-by: Ice3man <nizamulrana@gmail.com>
Co-authored-by: Nakul Bharti <knakul853@users.noreply.github.com>
* feat: fixed max-host-error blocking wrong port for template with error
* feat: log total results with time taken at end of execution
* bugfix: skip non-executed requests with progress in flow protocol
* feat: fixed request calculation in http protocol for progress
* misc adjustments
---------
Co-authored-by: Ice3man <nizamulrana@gmail.com>
* feat: global matchers
Signed-off-by: Dwi Siswanto <git@dw1.io>
Co-authored-by: Ice3man543 <ice3man543@users.noreply.github.com>
* feat(globalmatchers): make `Callback` as type
Signed-off-by: Dwi Siswanto <git@dw1.io>
* feat: update `passive` term to `(matchers-)static`
Signed-off-by: Dwi Siswanto <git@dw1.io>
* feat(globalmatchers): add `origin-template-*` event
also use `Set` method instead of `maps.Clone`
Signed-off-by: Dwi Siswanto <git@dw1.io>
* feat: update `matchers-static` term to `global-matchers`
Signed-off-by: Dwi Siswanto <git@dw1.io>
* feat(globalmatchers): clone event before `operator.Execute`
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(tmplexec): don't store `matched` on `global-matchers` templ
This will end up generating 2 events from the same
`scan.ScanContext` if one of the templates has
`global-matchers` enabled. This way, non-
`global-matchers` templates can enter the
`writeFailureCallback` func to log failure output.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* feat(globalmatchers): initializes `requests` on `New`
Signed-off-by: Dwi Siswanto <git@dw1.io>
* feat(globalmatchers): add `hasStorage` method
Signed-off-by: Dwi Siswanto <git@dw1.io>
* refactor(templates): rename global matchers checks method
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(loader): handle nil `templates.Template` pointer
Signed-off-by: Dwi Siswanto <git@dw1.io>
---------
Signed-off-by: Dwi Siswanto <git@dw1.io>
Co-authored-by: Ice3man543 <ice3man543@users.noreply.github.com>
* feat: conditionally panic-recover
As discussed with @Mzack9999, we should avoid
overusing panic-recover. We need to review the RCA
first to determine whether this is an exceptional
situation or if it's a higher-level function meant
to recover from a panic. This approach will help
us establish a robust error-handling strategy.
The implementation of panic-recover should be
conditional and NOT applied when running in a CI
environment AND IS temporary. Once we've caught
all errors and made the necessary corrections, we
can remove the deferred recover function.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* chore(deps): bump `go-ci` to v1.0.2
Signed-off-by: Dwi Siswanto <git@dw1.io>
* chore(make): add `-race` to `GOFLAGS` in `test`
Signed-off-by: Dwi Siswanto <git@dw1.io>
---------
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fail if OnResult callback is not called
* generate error message from error logs
* try..parse..
* fix lint
* add error message to last matcher event
* fix network protocol error logging
* log returned log from ExecuteWithResults
* add back specific logging
* clean up the msg
* minor
* init integration test for -ms
* add tests for http,network,js,ws protocols
* fix lint
* fix network test
* return err for dns protocol
* add integration test for dns protocol
* Apply input transformation to multi-protocol template execution
* Remove ad hoc input transoformation from DNS protocol
* Add SSL protocol input transformer
* Remove ad hoc input transoformation from SSL protocol
* Remove unused function extractDomain from the DNS protocol engine
* transform in flow as well
* bug fix + update test
* bug fix multi proto
:
* bug fix multi proto input
* bug fixes in input transform
---------
Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
* feat: added support for context cancellation to engine
* misc
* feat: added contexts everywhere
* misc
* misc
* use granular http timeouts and increase http timeout to 30s using multiplier
* track response header timeout in mhe
* update responseHeaderTimeout to 5sec
* skip failing windows test
---------
Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
* Fix panic err using flow templates with workflows
* Misc update
* skip test if pdcp keys are not present
---------
Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
* validate and fix empty internal-event
* fix on error with interactsh req
* disable clustering in flow & multiproto
* fix empty/missing matcher-status result
* fix cluster unit test
* fix no results found unit test
* handle 1 more edgecase
* add integration test for this edgecase
* fix multi-http-var-sharing with integration test
* add -payload-concurrency (-pc) flag
* fix missing internal:true login in multiprotocol engine
* fix/handle absolute invalid url parsing
* support -pc & -jc in go sdk
* fix missing variables in code protocol operators
* add payload count parallelhttp check
* feat http response memory optimization + reuse buffers
* update nuclei version
* feat: reuse js vm's and compile to programs
* fix failing http integration test
* remove dead code + add -jsc
* feat reuse js vms in pool with concurrency
* update comments as per review
* bug fix+ update interactsh test to look for dns interaction
* try enabling all interactsh integration tests
---------
Co-authored-by: mzack <marco.rivoli.nvh@gmail.com>
* log warnings + use scanContext in flow
* refactor flow to use scanContext + log all events
* feat: internal matcher
* fix integration test
* bug fix extractor: merge dynamic values, fix missing extractors in file
* flow: fix 'No Results Found' if last statement output is false
* fix unit test
* introduce `template-encoded` field
* remove IsCustomTemplate func
* refactor and move encoding to `MakeResultEventItem` func
* encode template in case of no results were found
* commit to last commit
* don't encode templates when`-ms` is used
* fix net read
* only return N bytes if extra available
* use ConnReadN from readerutil
* add integration test
* print unsigned warning in stderr
* fix js protocol in flow #4318
* fix integration test: url encoding issue
* fix network protocol issue + integration tests
* multiple improvements to integration test
* replace all conn.Read() from tests
* disable network-basic.yaml in windows
* disable code protocol in win CI
* fix bitwise login ps1-snippet.yaml
* hide previous matcher events in flow
* remove dead code+ update integration tests
---------
Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
