* Enhance matcher compilation with caching for regex and DSL expressions to improve performance. Update template parsing to conditionally retain raw templates based on size constraints.
* Implement caching for regex and DSL expressions in extractors and matchers to enhance performance. Introduce a buffer pool in raw requests to reduce memory allocations. Update template cache management for improved efficiency.
* feat: improve concurrency to be bound
* refactor: replace fmt.Sprintf with fmt.Fprintf for improved performance in header handling
* feat: add regex matching tests and benchmarks for performance evaluation
* feat: add prefix check in regex extraction to optimize matching process
* feat: implement regex caching mechanism to enhance performance in extractors and matchers, along with tests and benchmarks for validation
* feat: add unit tests for template execution in the core engine, enhancing test coverage and reliability
* feat: enhance error handling in template execution and improve regex caching logic for better performance
* Implement caching for regex and DSL expressions in the cache package, replacing previous sync.Map usage. Add unit tests for cache functionality, including eviction by capacity and retrieval of cached items. Update extractors and matchers to utilize the new cache system for improved performance and memory efficiency.
* Add tests for SetCapacities in cache package to ensure cache behavior on capacity changes
- Implemented TestSetCapacities_NoRebuildOnZero to verify that setting capacities to zero does not clear existing caches.
- Added TestSetCapacities_BeforeFirstUse to confirm that initial cache settings are respected and not overridden by subsequent capacity changes.
* Refactor matchers and update load test generator to use io package
- Removed maxRegexScanBytes constant from match.go.
- Replaced ioutil with io package in load_test.go for NopCloser usage.
- Restored TestValidate_AllowsInlineMultiline in load_test.go to ensure inline validation functionality.
* Add cancellation support in template execution and enhance test coverage
- Updated executeTemplateWithTargets to respect context cancellation.
- Introduced fakeTargetProvider and slowExecuter for testing.
- Added Test_executeTemplateWithTargets_RespectsCancellation to validate cancellation behavior during template execution.
* fix: remove undefined errorutil.ShowStackTrace
* feat: add make lint support and integrate with test
* refactor: migrate errorutil to errkit across codebase
- Replace deprecated errorutil with modern errkit
- Convert error declarations from var to func for better compatibility
- Fix all SA1019 deprecation warnings
- Maintain error chain support and stack traces
* fix: improve DNS test reliability using Google DNS
- Configure test to use Google DNS (8.8.8.8) for stability
- Fix nil pointer issue in DNS client initialization
- Keep production defaults unchanged
* fixing logic
* removing unwanted branches in makefile
---------
Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
* feat: fixed max-host-error blocking wrong port for template with error
* feat: log total results with time taken at end of execution
* bugfix: skip non-executed requests with progress in flow protocol
* feat: fixed request calculation in http protocol for progress
* misc adjustments
---------
Co-authored-by: Ice3man <nizamulrana@gmail.com>
This introduces a "nucleitcp" protocol that Nuclei will use when
making MySQL connections as part of its templates.
Previously, this would register (and de-register!) a custom "tcp"
dialer, and that applied globally, so any piece of software that
used a MySQL database and included nuclei in SDK mode would have
its database connections ripped out from under it due to the dialer
hijacking.
By using "nucleitcp" as the protocol, we are free to do whatever
we want with the dialer and not impact any other packages.
Within our `BuildDSN` function, we quietly replace the protocol to
"nucleitcp" if it was "tcp", so nuclei developers don't have to do
anything special to use this functionality; it will always do it.
* Move proxy variable from global to options
- Provides ability to pass diff proxy in single nuclei instance using sdk
* add type check (resolve comments)
* feat: global matchers
Signed-off-by: Dwi Siswanto <git@dw1.io>
Co-authored-by: Ice3man543 <ice3man543@users.noreply.github.com>
* feat(globalmatchers): make `Callback` as type
Signed-off-by: Dwi Siswanto <git@dw1.io>
* feat: update `passive` term to `(matchers-)static`
Signed-off-by: Dwi Siswanto <git@dw1.io>
* feat(globalmatchers): add `origin-template-*` event
also use `Set` method instead of `maps.Clone`
Signed-off-by: Dwi Siswanto <git@dw1.io>
* feat: update `matchers-static` term to `global-matchers`
Signed-off-by: Dwi Siswanto <git@dw1.io>
* feat(globalmatchers): clone event before `operator.Execute`
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(tmplexec): don't store `matched` on `global-matchers` templ
This will end up generating 2 events from the same
`scan.ScanContext` if one of the templates has
`global-matchers` enabled. This way, non-
`global-matchers` templates can enter the
`writeFailureCallback` func to log failure output.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* feat(globalmatchers): initializes `requests` on `New`
Signed-off-by: Dwi Siswanto <git@dw1.io>
* feat(globalmatchers): add `hasStorage` method
Signed-off-by: Dwi Siswanto <git@dw1.io>
* refactor(templates): rename global matchers checks method
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(loader): handle nil `templates.Template` pointer
Signed-off-by: Dwi Siswanto <git@dw1.io>
---------
Signed-off-by: Dwi Siswanto <git@dw1.io>
Co-authored-by: Ice3man543 <ice3man543@users.noreply.github.com>
* misc update
* chore(deps): bump github.com/gin-gonic/gin from 1.9.0 to 1.9.1 (#4252)
Bumps [github.com/gin-gonic/gin](https://github.com/gin-gonic/gin) from 1.9.0 to 1.9.1.
- [Release notes](https://github.com/gin-gonic/gin/releases)
- [Changelog](https://github.com/gin-gonic/gin/blob/master/CHANGELOG.md)
- [Commits](https://github.com/gin-gonic/gin/compare/v1.9.0...v1.9.1)
---
updated-dependencies:
- dependency-name: github.com/gin-gonic/gin
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(deps): bump github.com/docker/docker (#4316)
Bumps [github.com/docker/docker](https://github.com/docker/docker) from 24.0.5+incompatible to 24.0.7+incompatible.
- [Release notes](https://github.com/docker/docker/releases)
- [Commits](https://github.com/docker/docker/compare/v24.0.5...v24.0.7)
---
updated-dependencies:
- dependency-name: github.com/docker/docker
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix README_CN.md typos (#4369)
* version update
* Add more support for `fs.FS` in the disk catalog
This adds more support for `fs.FS` in the disk catalog. This
fixes some places where direct `os` file-related calls were being
made to use the catalog interface instead.
Note that the JavaScript compiler *still* does not work in any
context where the `pkg/js/libs/fs` package is used. In particular,
the `ReadFilesFromDir` function is hard-coded to use the `os`
package and not respect the catalog.
* Remove some testing artifacts
* Wrap up
* Unwind other changes
* Add a LoadHelperFileFunction to Options
* Use a direct func
* Tweak validation
* Use a function type
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xc1Ym <xuedongyuming2233@gmail.com>
Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
* Apply input transformation to multi-protocol template execution
* Remove ad hoc input transoformation from DNS protocol
* Add SSL protocol input transformer
* Remove ad hoc input transoformation from SSL protocol
* Remove unused function extractDomain from the DNS protocol engine
* transform in flow as well
* bug fix + update test
* bug fix multi proto
:
* bug fix multi proto input
* bug fixes in input transform
---------
Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
* feat: added fuzzing output enhancements
* changes as requested
* misc
* feat: added dfp flag to display fuzz points + misc additions
* feat: added support for fuzzing nested path segments
* feat: added parts to fuzzing requests
* feat: added tracking for parameter occurence frequency in fuzzing
* added cli flag for fuzz frequency
* fixed broken tests
* fixed path based sqli integration test
* feat: added configurable fuzzing aggression level for payloads
* fixed failing test
* feat: added support for context cancellation to engine
* misc
* feat: added contexts everywhere
* misc
* misc
* use granular http timeouts and increase http timeout to 30s using multiplier
* track response header timeout in mhe
* update responseHeaderTimeout to 5sec
* skip failing windows test
---------
Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
