This introduces a new Raw Data Export module to the codebase, enabling users to export raw log data via a dedicated API endpoint. The changes include the implementation of the module and handler, integration with existing infrastructure, configuration updates, and adjustments to tests and module wiring.
## 📄 Summary
To reliably migrate the alerts and dashboards, we need access to the telemetrystore to fetch some metadata and while doing migration, I need to log some stuff to fix stuff later.
Key changes:
- Modified the migration to include telemetrystore and a logging provider (open to using a standard logger instead)
- To avoid the previous issues with imported dashboards failing during migration, I've ensured that imported JSON files are automatically transformed when migration is active
- Implemented detailed logic to handle dashboard migration cleanly and prevent unnecessary errors
- Separated the core migration logic from SQL migration code, as users from the dot metrics migration requested shareable code snippets for local migrations. This modular approach allows others to easily reuse the migration functionality.
Known: I didn't register the migration yet in this PR, and will not merge this yet, so please review with that in mid.
Deprecate all flags
- Use querier.config.fluxInterval in lieu of passing `--flux-interval` and `--flux-interval-for-trace-detail`
- Remove `--gateway-url`
- Use telemetrystore.clickhouse.cluster in lieu of passing `--cluster` or `--cluster-name`
- Add an `unparam` check in the linter. Updated some functions across the querier codebase to be compatible with this linter.
- Remove prometheus config from docker builds.
* feat(user): support sso and api key
* feat(user): remove ee references from pkg
* feat(user): remove ee references from pkg
* feat(user): related client changes
* feat(user): remove the sso available check
* feat(user): fix go tests
* feat(user): move the middleware from ee to pkg
* feat(user): some more error code cleanup
* feat(user): some more error code cleanup
* feat(user): skip flaky UI tests
* feat(user): some more error code cleanup
* chore(savedview): refactor into module and handler
* chore(rule): move telemetry inside telemetry
* chore(apdex): refactor apdex and delete dao
* chore(dashboard): create a dashboard module
* chore(ee): get rid of the init nonesense
* chore(dashboard): fix err and apierror confusion
* chore: address comments
* feat(organization): add hname and alias for organization
* fix: boolean values are not shown in the list panel's column
* fix: moved logic to component level
* fix: added type
* fix: added test cases
* fix: added test cases
* chore: update copy webpack plugin
* Revert "fix: display same key with multiple data types in filter suggestions by enhancing the deduping logic (#7255)"
This reverts commit 1e85981a17a8e715e948308d3e85072d976907d3.
* fix: use query search v2 for traces data source to handle multiple data types for the same key
* fix(QueryBuilderSearchV2): add user typed option if it doesn't exist in the payload
* fix(QueryBuilderSearchV2): increase the height of search dropdown for non-logs data sources
* fix: display span scope selector for trace data source
* chore: remove the span scope selector from qb search v1 and move the component to search v2
* fix: write test to ensure that we display span scope selector for traces data source
* fix: limit converting -> only to log data source
* fix: don't display empty suggestion if only spaces are typed
* chore: tests for span scope selector
* chore: qb search flow (key, operator, value) test cases
* refactor: fix the Maximum update depth reached issue while running tests
* chore: overall improvements to span scope selector tests
Resource attr filter: style fix and quick filter changes (#7691)
* chore: resource attr filter init
* chore: resource attr filter api integration
* chore: operator config updated
* chore: fliter show hide logic and styles
* chore: add support for custom operator list to qb
* chore: minor refactor
* chore: minor code refactor
* test: quick filters test suite added
* test: quick filters test suite added
* test: all errors test suite added
* chore: style fix
* test: all errors mock fix
* chore: test case fix and mixpanel update
* chore: color update
* chore: minor refactor
* chore: style fix
* chore: set default query in exceptions tab
* chore: style fix
* chore: minor refactor
* chore: minor refactor
* chore: minor refactor
* chore: test update
* chore: fix filter header with no query name
* fix: scroll fix
* chore: add data source traces to quick filters
* chore: replace div with fragment
---------
Co-authored-by: Aditya Singh <adityasingh@Adityas-MacBook-Pro.local>
fix: handle rate operators for table panel (#7695)
* fix: handle rate operators for table panel
chore: fix error rate (#7701)
Signed-off-by: Shivanshu Raj Shrivastava <shivanshu1333@gmail.com>
* feat(organization): minor cleanups
* feat(organization): better naming for api and usecase
* feat(organization): better packaging for modules
* feat(organization): change hname to displayName
* feat(organization): update the migration to use dialect
* feat(organization): update the migration to use dialect
* feat(organization): update the migration to use dialect
* feat(organization): revert back to impl
* feat(organization): remove DI from organization
* feat(organization): address review comments
* feat(organization): address review comments
* feat(organization): address review comments
---------
Signed-off-by: Shivanshu Raj Shrivastava <shivanshu1333@gmail.com>
* fix: support multitenancy in org
* fix: register and login working now
* fix: changes to migration
* fix: migrations run both on sqlite and postgres
* fix: remove user flags from fe and be
* fix: remove ingestion keys from update
* fix: multitenancy support for apdex settings
* fix: render ts for users correctly
* fix: fix migration to run for new tenants
* fix: clean up migrations
* fix: address comments
* Update pkg/sqlmigration/013_update_organization.go
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* fix: fix build
* fix: force invites with org id
* Update pkg/query-service/auth/auth.go
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* fix: address comments
* fix: address comments
* fix: provier with their own dialect
* fix: update dialects
* fix: remove unwanted change
* Update pkg/query-service/app/http_handler.go
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* fix: different files for types
---------
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* chore: add test validating invalid field paths in pipeline operators are rejected
* chore: refactor posted pipelines validation to use a controller method
* fix: run a collector simulation to validate pipeline config being saved
* chore: minor cleanup
* chore: stash initial work with API signature
* chore: put together setup for integration testing filter suggestions
* feat: filter suggestions: suggest attribs using existing autocomplete logic
* chore: filter suggestions test: add expectation for example queries
* feat: filter suggestions: default suggestions when data yet to be received
* feat: finish plumbing basic example queries
* chore: add test for filter suggestions with an existing query
* feat: filter suggestions: don't suggest attribs already included in existing filter
* chore: generate example queries by including existing filter first
* chore: upgrade ClickHouse-go-mock
* chore: some cleanup of reader.GetQBFilterSuggestionsForLogs
* chore: some cleanup of filter suggestion tests
* chore: some cleanup to http handler and request parsing logic for filter suggestions
* chore: remove expectation that attrib suggestions won't contain attribs already used in filter
* chore: refactor: inject sqlx.DB into opamp.initDB instead of DB file name
* chore: reorganize test utils a little
* chore: add test validating pipelines for installed integrations show up in pipelines list
* chore: get basic integration pipelines testcase passing
* chore: reconcile experimental changes with latest state of develop
* chore: add integration test for reordering of pipelines
* chore: marker for integration pipelines using Id
* chore: hookup propagation of installed integration pipelines by opamp
* chore: add util for mapping slices
* chore: add support for reordering integration pipelines
* chore: exclude user saved integration pipelines if no longer installed
* chore: flesh out rest of intgeration pipelines scenarios
* chore: handle scenario when an integration is installed before any pipelines exist
* chore: notify agentConf of update after uninstalling an integration
* chore: some minor cleanup
* chore: some more cleanup
* chore: update ee server for changed controllers
* chore: some more cleanup
* chore: change builtin integration id prefix to avoid using colons that break yaml
* chore: update builtin integration id in test
* chore: add http api test for signoz integrations
* chore: add controller for integrations
* chore: add http API handlers for integrations API
* chore: hook up integrations API in new servers
* chore: add remaining fields in Integration DTO
---------
Co-authored-by: Srikanth Chekuri <srikanth.chekuri92@gmail.com>
* fix: improve user telemetry
- move GetEmailFromJwt to common function
- update AttachJwtToContext() to use standard way of attaching value to context
- update userEmail in every possible sendEvent call
- send groupId in sendEvent call for SaaS operator analytics
* chore: added DEFAULT_CLOUD_EMAIL const
* chore: add AttachJwtToContext to analytics middleware
* test: added AttachJwtToContext to logs pipelines
* chore: add integration test for log pipelines being recommended to agents on connection
* chore: agentConf.Manager as AgentConfigProvider
* feat: logparsingpipelines as an AgentFeature
* chore: some cleanup
* feat: update agentConf init in ee query-service
* chore: more cleanup and move opamp/logspipeline -> logparsingpipeline/collector_config.go
* chore: some more cleanup
* chore: set agent.RemoteConfig after broadcasting conf to all agents
* chore: add test scenario for post remote conf application
* feat: add interface for opamp.AgentConfigProvider
* feat: add iface and plumbing for generating recommended conf in opamp/agent
* feat: get opamp server config provider tests started
* chore: add test scenario for agent connection without a config recommendation
* chore: add test scenario for agent connection with a config recommendation
* chore: add test for validating config deployment status gets reported
* chore: add test for rolling out latest config recommendations when config changes
* chore: wrap up opamp server lifecycle tests
* chore: some tests cleanup
* chore: get all tests passing
* chore: update opamp server init logic in ee query service
* chore: some cleanup
* chore: some final cleanup
* chore: use v3.Filterset as pipeline filters in logparsing pipelines integration tests
* chore: get logparsing integration tests passing with filterset based pipeline
* chore: get all other breaking tests passing
* chore: move models.logparsingpipeline to logparsingpipeline.model
* chore: implement Valuer and Scanner interfaces for v3.FilterSet
* chore: add integration test for log parsing pipelines validation
* chore: add helpers for creating unauthorized, unavailable & not found api errors
* chore: return *model.APIError from logpipeline and agentConf functions
* chore: some cleanup
* chore: some more cleanup
* chore: one more round of cleanups
* fix: avoid pipeline builder panic if first op is disabled
* chore: use pipeline builder test to trigger getOperator panic
---------
Co-authored-by: Nityananda Gohain <nityanandagohain@gmail.com>
