This introduces a new Raw Data Export module to the codebase, enabling users to export raw log data via a dedicated API endpoint. The changes include the implementation of the module and handler, integration with existing infrastructure, configuration updates, and adjustments to tests and module wiring.
## 📄 Summary
To reliably migrate the alerts and dashboards, we need access to the telemetrystore to fetch some metadata and while doing migration, I need to log some stuff to fix stuff later.
Key changes:
- Modified the migration to include telemetrystore and a logging provider (open to using a standard logger instead)
- To avoid the previous issues with imported dashboards failing during migration, I've ensured that imported JSON files are automatically transformed when migration is active
- Implemented detailed logic to handle dashboard migration cleanly and prevent unnecessary errors
- Separated the core migration logic from SQL migration code, as users from the dot metrics migration requested shareable code snippets for local migrations. This modular approach allows others to easily reuse the migration functionality.
Known: I didn't register the migration yet in this PR, and will not merge this yet, so please review with that in mid.
Deprecate all flags
- Use querier.config.fluxInterval in lieu of passing `--flux-interval` and `--flux-interval-for-trace-detail`
- Remove `--gateway-url`
- Use telemetrystore.clickhouse.cluster in lieu of passing `--cluster` or `--cluster-name`
- Add an `unparam` check in the linter. Updated some functions across the querier codebase to be compatible with this linter.
- Remove prometheus config from docker builds.
* feat(user): support sso and api key
* feat(user): remove ee references from pkg
* feat(user): remove ee references from pkg
* feat(user): related client changes
* feat(user): remove the sso available check
* feat(user): fix go tests
* feat(user): move the middleware from ee to pkg
* feat(user): some more error code cleanup
* feat(user): some more error code cleanup
* feat(user): skip flaky UI tests
* feat(user): some more error code cleanup
* chore(linter): add more linters and deprecate zap
* chore(linter): add more linters and deprecate zap
* chore(linter): add more linters and deprecate zap
* chore(linter): add more linters and deprecate zap
* chore(savedview): refactor into module and handler
* chore(rule): move telemetry inside telemetry
* chore(apdex): refactor apdex and delete dao
* chore(dashboard): create a dashboard module
* chore(ee): get rid of the init nonesense
* chore(dashboard): fix err and apierror confusion
* chore: address comments
* feat: added changes related to custom options for quick filters
* feat: added changes related to custom options for quick filters
* feat: added changes related to custom options for quick filters
* feat: added changes related to custom options for quick filters
* feat: added changes related to custom options for quick filters
* feat: added changes related to custom options for quick filters
* feat: added support for custom quick filters
* feat: added changes related to custom options for quick filters
* feat: added changes related to custom options for quick filters
* feat: added changes related to custom options for quick filters
* feat: added changes related to custom options for quick filters
* feat: added changes related to custom options for quick filters
* feat: added changes related to custom options for quick filters
* feat: added changes related to custom options for quick filters
* feat: added changes related to custom options for quick filters
* feat(organization): add hname and alias for organization
* fix: boolean values are not shown in the list panel's column
* fix: moved logic to component level
* fix: added type
* fix: added test cases
* fix: added test cases
* chore: update copy webpack plugin
* Revert "fix: display same key with multiple data types in filter suggestions by enhancing the deduping logic (#7255)"
This reverts commit 1e85981a17a8e715e948308d3e85072d976907d3.
* fix: use query search v2 for traces data source to handle multiple data types for the same key
* fix(QueryBuilderSearchV2): add user typed option if it doesn't exist in the payload
* fix(QueryBuilderSearchV2): increase the height of search dropdown for non-logs data sources
* fix: display span scope selector for trace data source
* chore: remove the span scope selector from qb search v1 and move the component to search v2
* fix: write test to ensure that we display span scope selector for traces data source
* fix: limit converting -> only to log data source
* fix: don't display empty suggestion if only spaces are typed
* chore: tests for span scope selector
* chore: qb search flow (key, operator, value) test cases
* refactor: fix the Maximum update depth reached issue while running tests
* chore: overall improvements to span scope selector tests
Resource attr filter: style fix and quick filter changes (#7691)
* chore: resource attr filter init
* chore: resource attr filter api integration
* chore: operator config updated
* chore: fliter show hide logic and styles
* chore: add support for custom operator list to qb
* chore: minor refactor
* chore: minor code refactor
* test: quick filters test suite added
* test: quick filters test suite added
* test: all errors test suite added
* chore: style fix
* test: all errors mock fix
* chore: test case fix and mixpanel update
* chore: color update
* chore: minor refactor
* chore: style fix
* chore: set default query in exceptions tab
* chore: style fix
* chore: minor refactor
* chore: minor refactor
* chore: minor refactor
* chore: test update
* chore: fix filter header with no query name
* fix: scroll fix
* chore: add data source traces to quick filters
* chore: replace div with fragment
---------
Co-authored-by: Aditya Singh <adityasingh@Adityas-MacBook-Pro.local>
fix: handle rate operators for table panel (#7695)
* fix: handle rate operators for table panel
chore: fix error rate (#7701)
Signed-off-by: Shivanshu Raj Shrivastava <shivanshu1333@gmail.com>
* feat(organization): minor cleanups
* feat(organization): better naming for api and usecase
* feat(organization): better packaging for modules
* feat(organization): change hname to displayName
* feat(organization): update the migration to use dialect
* feat(organization): update the migration to use dialect
* feat(organization): update the migration to use dialect
* feat(organization): revert back to impl
* feat(organization): remove DI from organization
* feat(organization): address review comments
* feat(organization): address review comments
* feat(organization): address review comments
---------
Signed-off-by: Shivanshu Raj Shrivastava <shivanshu1333@gmail.com>
* feat(sqlmigration): update the alertmanager tables
* feat(sqlmigration): update the alertmanager tables
* feat(sqlmigration): make the preference package multi tenant
* feat(preference): address nit pick comments
* feat(preference): added the cascade delete for preferences
* feat(sqlmigration): update apdex and TTL status tables (#7481)
* feat(sqlmigration): update the apdex and ttl tables
* feat(sqlmigration): register the new migration and rename table
* feat(sqlmigration): fix the ttl queries
* feat(sqlmigration): update the TTL and apdex tables
* feat(sqlmigration): update the TTL and apdex tables
* fix: support multitenancy in dashboards
* fix: support multitenancy in saved views
* fix: move migrations to provider file
* fix: remove getUserFromClaims and use new errors
* fix: remove getUserFromClaims and use new errors
* fix: use new errors in dashboards.go
* Update ee/query-service/app/api/dashboard.go
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* fix: minor changes
---------
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* fix: support multitenancy in org
* fix: register and login working now
* fix: changes to migration
* fix: migrations run both on sqlite and postgres
* fix: remove user flags from fe and be
* fix: remove ingestion keys from update
* fix: multitenancy support for apdex settings
* fix: render ts for users correctly
* fix: fix migration to run for new tenants
* fix: clean up migrations
* fix: address comments
* Update pkg/sqlmigration/013_update_organization.go
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* fix: fix build
* fix: force invites with org id
* Update pkg/query-service/auth/auth.go
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* fix: address comments
* fix: address comments
* fix: provier with their own dialect
* fix: update dialects
* fix: remove unwanted change
* Update pkg/query-service/app/http_handler.go
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* fix: different files for types
---------
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* chore: get started on dashboards plumbing for AWS integration services
* feat: cloud integrations: include cloud integrations dashboards in dashboards list
* feat: cloud integrations: get cloud integration dashboard by id
* feat: dashboard url in cloud integrations svc detail
* feat: ec2 overview dashboard
* chore: add ec2 overview dashboard image
* chore: finish up with v0 definitions for EC2 and RDS
* chore: some cleanup
* chore: fix broken test
* chore: add connection url composition
---------
Co-authored-by: Srikanth Chekuri <srikanth.chekuri92@gmail.com>
* chore: stash initial work with API signature
* chore: put together setup for integration testing filter suggestions
* feat: filter suggestions: suggest attribs using existing autocomplete logic
* chore: filter suggestions test: add expectation for example queries
* feat: filter suggestions: default suggestions when data yet to be received
* feat: finish plumbing basic example queries
* chore: add test for filter suggestions with an existing query
* feat: filter suggestions: don't suggest attribs already included in existing filter
* chore: generate example queries by including existing filter first
* chore: upgrade ClickHouse-go-mock
* chore: some cleanup of reader.GetQBFilterSuggestionsForLogs
* chore: some cleanup of filter suggestion tests
* chore: some cleanup to http handler and request parsing logic for filter suggestions
* chore: remove expectation that attrib suggestions won't contain attribs already used in filter
* chore: add test for updatedAt value being populated in integration dashboards and get it passing
* chore: also populate createdAt, createBy and updateBy for instaled integration dashboards
* chore: update clickhouse integration config instructions
* feat: flesh out pre-requisites for collecting mongodb logs and metrics
* chore: remove stale pipelines in bundled integrations
* chore: clean up 'collect metrics' step for mongodb
* feat: add instructions for collecting and parsing mongodb logs
* feat: add metrics and logs attributes to mongodb data collected list
* feat: nginx logs collection instructions and some other cleanup
* feat: add list of parsed log attributes to data collected list for nginx
* chore: do not run pipeline population integration test if no built-in integration has a pipeline
* chore: add test expectations for integration metrics connection status
* chore: reorg logs connection status calculation for parallelization
* chore: add interface for reader.GetMetricLastReceivedTsMillis
* chore: add plumbing for calculating integration metrics connection status
* chore: impl and test mocks for reader.GetMetricReceivedLatest
* chore: wrap things up and get test passing
* chore: some cleanup
* chore: some more cleanup
* chore: use prom metric names for integration connection test
* chore: add test for dashboards for installed integrations
* feat: include dashboards for installed integrations in API response
* chore: add test expectation for getting installed integration dashboard by id
* feat: add support for retrieving installed integration dashboards by id
* chore: add dashboard id validation for integrations
* chore: refactor: inject sqlx.DB into opamp.initDB instead of DB file name
* chore: reorganize test utils a little
* chore: add test validating pipelines for installed integrations show up in pipelines list
* chore: get basic integration pipelines testcase passing
* chore: reconcile experimental changes with latest state of develop
* chore: add integration test for reordering of pipelines
* chore: marker for integration pipelines using Id
* chore: hookup propagation of installed integration pipelines by opamp
* chore: add util for mapping slices
* chore: add support for reordering integration pipelines
* chore: exclude user saved integration pipelines if no longer installed
* chore: flesh out rest of intgeration pipelines scenarios
* chore: handle scenario when an integration is installed before any pipelines exist
* chore: notify agentConf of update after uninstalling an integration
* chore: some minor cleanup
* chore: some more cleanup
* chore: update ee server for changed controllers
* chore: some more cleanup
* chore: change builtin integration id prefix to avoid using colons that break yaml
* chore: update builtin integration id in test
* chore: add integration attribs for connection tests and status
* chore: add connection status to integration details response
* chore: update integration lifecycle test to check for connection status too
* feat: add GetIntegrationConnectionTests to integrations manager and controller
* chore: add tests for querying integration connection status
* feat: add http API support for integration connection status
* chore: some cleanups
* chore: use PostableRule for integration alerts
* chore: some more cleanup
* chore: add http api test for signoz integrations
* chore: add controller for integrations
* chore: add http API handlers for integrations API
* chore: hook up integrations API in new servers
* chore: add remaining fields in Integration DTO
---------
Co-authored-by: Srikanth Chekuri <srikanth.chekuri92@gmail.com>
