71 lines
1.7 KiB
Go
Raw Normal View History

JSON Export Handling Updates (#3466) * Switch -json to -jsonl * Add JSON output file * Update docs for EN and ID * Fix linting issue with error wrap * Add -j flag * Fix call for short flag * Correct typo "Ciper" to "Cipher" (#3468) * migrate dsl helper functions to dsl repo (#3461) * migrate dsl pkg code to dsl repo * fix lint error * upgrade dsl dependency * upgrade deps --------- Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io> * chore(deps): bump github.com/projectdiscovery/httpx in /v2 (#3469) Bumps [github.com/projectdiscovery/httpx](https://github.com/projectdiscovery/httpx) from 1.2.7 to 1.2.9. - [Release notes](https://github.com/projectdiscovery/httpx/releases) - [Changelog](https://github.com/projectdiscovery/httpx/blob/main/.goreleaser.yml) - [Commits](https://github.com/projectdiscovery/httpx/compare/v1.2.7...v1.2.9) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/httpx dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump github.com/weppos/publicsuffix-go in /v2 (#3472) Bumps [github.com/weppos/publicsuffix-go](https://github.com/weppos/publicsuffix-go) from 0.20.0 to 0.30.0. - [Release notes](https://github.com/weppos/publicsuffix-go/releases) - [Changelog](https://github.com/weppos/publicsuffix-go/blob/main/CHANGELOG.md) - [Commits](https://github.com/weppos/publicsuffix-go/compare/v0.20.0...v0.30.0) --- updated-dependencies: - dependency-name: github.com/weppos/publicsuffix-go dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump github.com/projectdiscovery/wappalyzergo in /v2 (#3473) Bumps [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) from 0.0.81 to 0.0.88. - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.0.81...v0.0.88) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump github.com/projectdiscovery/hmap in /v2 (#3470) Bumps [github.com/projectdiscovery/hmap](https://github.com/projectdiscovery/hmap) from 0.0.10 to 0.0.11. - [Release notes](https://github.com/projectdiscovery/hmap/releases) - [Commits](https://github.com/projectdiscovery/hmap/compare/v0.0.10...v0.0.11) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/hmap dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * debug catalog path * use paths instead of filepath for aws path * deps update (#3477) * deps update * fixing gologger via callback * Moved `json-export` flag to the other exporters * Switch "json[-_]exporter to jsonexporter" --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Ramana Reddy <90540245+RamanaReddy0M@users.noreply.github.com> Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io> Co-authored-by: Mzack9999 <mzack9999@protonmail.com> Co-authored-by: shubhamrasal <shubhamdharmarasal@gmail.com>
2023-03-31 05:59:29 -04:00
package jsonexporter
import (
"os"
"sync"
"github.com/pkg/errors"
"github.com/projectdiscovery/nuclei/v3/pkg/output"
"github.com/projectdiscovery/nuclei/v3/pkg/utils/json"
JSON Export Handling Updates (#3466) * Switch -json to -jsonl * Add JSON output file * Update docs for EN and ID * Fix linting issue with error wrap * Add -j flag * Fix call for short flag * Correct typo "Ciper" to "Cipher" (#3468) * migrate dsl helper functions to dsl repo (#3461) * migrate dsl pkg code to dsl repo * fix lint error * upgrade dsl dependency * upgrade deps --------- Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io> * chore(deps): bump github.com/projectdiscovery/httpx in /v2 (#3469) Bumps [github.com/projectdiscovery/httpx](https://github.com/projectdiscovery/httpx) from 1.2.7 to 1.2.9. - [Release notes](https://github.com/projectdiscovery/httpx/releases) - [Changelog](https://github.com/projectdiscovery/httpx/blob/main/.goreleaser.yml) - [Commits](https://github.com/projectdiscovery/httpx/compare/v1.2.7...v1.2.9) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/httpx dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump github.com/weppos/publicsuffix-go in /v2 (#3472) Bumps [github.com/weppos/publicsuffix-go](https://github.com/weppos/publicsuffix-go) from 0.20.0 to 0.30.0. - [Release notes](https://github.com/weppos/publicsuffix-go/releases) - [Changelog](https://github.com/weppos/publicsuffix-go/blob/main/CHANGELOG.md) - [Commits](https://github.com/weppos/publicsuffix-go/compare/v0.20.0...v0.30.0) --- updated-dependencies: - dependency-name: github.com/weppos/publicsuffix-go dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump github.com/projectdiscovery/wappalyzergo in /v2 (#3473) Bumps [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) from 0.0.81 to 0.0.88. - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.0.81...v0.0.88) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump github.com/projectdiscovery/hmap in /v2 (#3470) Bumps [github.com/projectdiscovery/hmap](https://github.com/projectdiscovery/hmap) from 0.0.10 to 0.0.11. - [Release notes](https://github.com/projectdiscovery/hmap/releases) - [Commits](https://github.com/projectdiscovery/hmap/compare/v0.0.10...v0.0.11) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/hmap dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * debug catalog path * use paths instead of filepath for aws path * deps update (#3477) * deps update * fixing gologger via callback * Moved `json-export` flag to the other exporters * Switch "json[-_]exporter to jsonexporter" --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Ramana Reddy <90540245+RamanaReddy0M@users.noreply.github.com> Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io> Co-authored-by: Mzack9999 <mzack9999@protonmail.com> Co-authored-by: shubhamrasal <shubhamdharmarasal@gmail.com>
2023-03-31 05:59:29 -04:00
)
type Exporter struct {
options *Options
mutex *sync.Mutex
rows []output.ResultEvent
}
// Options contains the configuration options for JSON exporter client
type Options struct {
// File is the file to export found JSON result to
File string `yaml:"file"`
OmitRaw bool `yaml:"omit-raw"`
JSON Export Handling Updates (#3466) * Switch -json to -jsonl * Add JSON output file * Update docs for EN and ID * Fix linting issue with error wrap * Add -j flag * Fix call for short flag * Correct typo "Ciper" to "Cipher" (#3468) * migrate dsl helper functions to dsl repo (#3461) * migrate dsl pkg code to dsl repo * fix lint error * upgrade dsl dependency * upgrade deps --------- Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io> * chore(deps): bump github.com/projectdiscovery/httpx in /v2 (#3469) Bumps [github.com/projectdiscovery/httpx](https://github.com/projectdiscovery/httpx) from 1.2.7 to 1.2.9. - [Release notes](https://github.com/projectdiscovery/httpx/releases) - [Changelog](https://github.com/projectdiscovery/httpx/blob/main/.goreleaser.yml) - [Commits](https://github.com/projectdiscovery/httpx/compare/v1.2.7...v1.2.9) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/httpx dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump github.com/weppos/publicsuffix-go in /v2 (#3472) Bumps [github.com/weppos/publicsuffix-go](https://github.com/weppos/publicsuffix-go) from 0.20.0 to 0.30.0. - [Release notes](https://github.com/weppos/publicsuffix-go/releases) - [Changelog](https://github.com/weppos/publicsuffix-go/blob/main/CHANGELOG.md) - [Commits](https://github.com/weppos/publicsuffix-go/compare/v0.20.0...v0.30.0) --- updated-dependencies: - dependency-name: github.com/weppos/publicsuffix-go dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump github.com/projectdiscovery/wappalyzergo in /v2 (#3473) Bumps [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) from 0.0.81 to 0.0.88. - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.0.81...v0.0.88) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump github.com/projectdiscovery/hmap in /v2 (#3470) Bumps [github.com/projectdiscovery/hmap](https://github.com/projectdiscovery/hmap) from 0.0.10 to 0.0.11. - [Release notes](https://github.com/projectdiscovery/hmap/releases) - [Commits](https://github.com/projectdiscovery/hmap/compare/v0.0.10...v0.0.11) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/hmap dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * debug catalog path * use paths instead of filepath for aws path * deps update (#3477) * deps update * fixing gologger via callback * Moved `json-export` flag to the other exporters * Switch "json[-_]exporter to jsonexporter" --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Ramana Reddy <90540245+RamanaReddy0M@users.noreply.github.com> Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io> Co-authored-by: Mzack9999 <mzack9999@protonmail.com> Co-authored-by: shubhamrasal <shubhamdharmarasal@gmail.com>
2023-03-31 05:59:29 -04:00
}
// New creates a new JSON exporter integration client based on options.
func New(options *Options) (*Exporter, error) {
exporter := &Exporter{
mutex: &sync.Mutex{},
options: options,
rows: []output.ResultEvent{},
}
return exporter, nil
}
// Export appends the passed result event to the list of objects to be exported to
// the resulting JSON file
func (exporter *Exporter) Export(event *output.ResultEvent) error {
exporter.mutex.Lock()
defer exporter.mutex.Unlock()
if exporter.options.OmitRaw {
event.Request = ""
event.Response = ""
}
JSON Export Handling Updates (#3466) * Switch -json to -jsonl * Add JSON output file * Update docs for EN and ID * Fix linting issue with error wrap * Add -j flag * Fix call for short flag * Correct typo "Ciper" to "Cipher" (#3468) * migrate dsl helper functions to dsl repo (#3461) * migrate dsl pkg code to dsl repo * fix lint error * upgrade dsl dependency * upgrade deps --------- Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io> * chore(deps): bump github.com/projectdiscovery/httpx in /v2 (#3469) Bumps [github.com/projectdiscovery/httpx](https://github.com/projectdiscovery/httpx) from 1.2.7 to 1.2.9. - [Release notes](https://github.com/projectdiscovery/httpx/releases) - [Changelog](https://github.com/projectdiscovery/httpx/blob/main/.goreleaser.yml) - [Commits](https://github.com/projectdiscovery/httpx/compare/v1.2.7...v1.2.9) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/httpx dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump github.com/weppos/publicsuffix-go in /v2 (#3472) Bumps [github.com/weppos/publicsuffix-go](https://github.com/weppos/publicsuffix-go) from 0.20.0 to 0.30.0. - [Release notes](https://github.com/weppos/publicsuffix-go/releases) - [Changelog](https://github.com/weppos/publicsuffix-go/blob/main/CHANGELOG.md) - [Commits](https://github.com/weppos/publicsuffix-go/compare/v0.20.0...v0.30.0) --- updated-dependencies: - dependency-name: github.com/weppos/publicsuffix-go dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump github.com/projectdiscovery/wappalyzergo in /v2 (#3473) Bumps [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) from 0.0.81 to 0.0.88. - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.0.81...v0.0.88) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump github.com/projectdiscovery/hmap in /v2 (#3470) Bumps [github.com/projectdiscovery/hmap](https://github.com/projectdiscovery/hmap) from 0.0.10 to 0.0.11. - [Release notes](https://github.com/projectdiscovery/hmap/releases) - [Commits](https://github.com/projectdiscovery/hmap/compare/v0.0.10...v0.0.11) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/hmap dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * debug catalog path * use paths instead of filepath for aws path * deps update (#3477) * deps update * fixing gologger via callback * Moved `json-export` flag to the other exporters * Switch "json[-_]exporter to jsonexporter" --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Ramana Reddy <90540245+RamanaReddy0M@users.noreply.github.com> Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io> Co-authored-by: Mzack9999 <mzack9999@protonmail.com> Co-authored-by: shubhamrasal <shubhamdharmarasal@gmail.com>
2023-03-31 05:59:29 -04:00
// Add the event to the rows
exporter.rows = append(exporter.rows, *event)
return nil
}
// Close writes the in-memory data to the JSON file specified by options.JSONExport
// and closes the exporter after operation
func (exporter *Exporter) Close() error {
exporter.mutex.Lock()
defer exporter.mutex.Unlock()
// Convert the rows to JSON byte array
obj, err := json.Marshal(exporter.rows)
if err != nil {
return errors.Wrap(err, "failed to generate JSON report")
}
// Attempt to write the JSON to file specified in options.JSONExport
if err := os.WriteFile(exporter.options.File, obj, 0644); err != nil {
return errors.Wrap(err, "failed to create JSON file")
}
return nil
}