## 📄 Summary
To reliably migrate the alerts and dashboards, we need access to the telemetrystore to fetch some metadata and while doing migration, I need to log some stuff to fix stuff later.
Key changes:
- Modified the migration to include telemetrystore and a logging provider (open to using a standard logger instead)
- To avoid the previous issues with imported dashboards failing during migration, I've ensured that imported JSON files are automatically transformed when migration is active
- Implemented detailed logic to handle dashboard migration cleanly and prevent unnecessary errors
- Separated the core migration logic from SQL migration code, as users from the dot metrics migration requested shareable code snippets for local migrations. This modular approach allows others to easily reuse the migration functionality.
Known: I didn't register the migration yet in this PR, and will not merge this yet, so please review with that in mid.
Deprecate all flags
- Use querier.config.fluxInterval in lieu of passing `--flux-interval` and `--flux-interval-for-trace-detail`
- Remove `--gateway-url`
- Use telemetrystore.clickhouse.cluster in lieu of passing `--cluster` or `--cluster-name`
- Add an `unparam` check in the linter. Updated some functions across the querier codebase to be compatible with this linter.
- Remove prometheus config from docker builds.
* feat(user): support sso and api key
* feat(user): remove ee references from pkg
* feat(user): remove ee references from pkg
* feat(user): related client changes
* feat(user): remove the sso available check
* feat(user): fix go tests
* feat(user): move the middleware from ee to pkg
* feat(user): some more error code cleanup
* feat(user): some more error code cleanup
* feat(user): skip flaky UI tests
* feat(user): some more error code cleanup
* chore(linter): add more linters and deprecate zap
* chore(linter): add more linters and deprecate zap
* chore(linter): add more linters and deprecate zap
* chore(linter): add more linters and deprecate zap
* chore(savedview): refactor into module and handler
* chore(rule): move telemetry inside telemetry
* chore(apdex): refactor apdex and delete dao
* chore(dashboard): create a dashboard module
* chore(ee): get rid of the init nonesense
* chore(dashboard): fix err and apierror confusion
* chore: address comments
* feat: added changes related to custom options for quick filters
* feat: added changes related to custom options for quick filters
* feat: added changes related to custom options for quick filters
* feat: added changes related to custom options for quick filters
* feat: added changes related to custom options for quick filters
* feat: added changes related to custom options for quick filters
* feat: added support for custom quick filters
* feat: added changes related to custom options for quick filters
* feat: added changes related to custom options for quick filters
* feat: added changes related to custom options for quick filters
* feat: added changes related to custom options for quick filters
* feat: added changes related to custom options for quick filters
* feat: added changes related to custom options for quick filters
* feat: added changes related to custom options for quick filters
* feat: added changes related to custom options for quick filters
* feat(organization): add hname and alias for organization
* fix: boolean values are not shown in the list panel's column
* fix: moved logic to component level
* fix: added type
* fix: added test cases
* fix: added test cases
* chore: update copy webpack plugin
* Revert "fix: display same key with multiple data types in filter suggestions by enhancing the deduping logic (#7255)"
This reverts commit 1e85981a17a8e715e948308d3e85072d976907d3.
* fix: use query search v2 for traces data source to handle multiple data types for the same key
* fix(QueryBuilderSearchV2): add user typed option if it doesn't exist in the payload
* fix(QueryBuilderSearchV2): increase the height of search dropdown for non-logs data sources
* fix: display span scope selector for trace data source
* chore: remove the span scope selector from qb search v1 and move the component to search v2
* fix: write test to ensure that we display span scope selector for traces data source
* fix: limit converting -> only to log data source
* fix: don't display empty suggestion if only spaces are typed
* chore: tests for span scope selector
* chore: qb search flow (key, operator, value) test cases
* refactor: fix the Maximum update depth reached issue while running tests
* chore: overall improvements to span scope selector tests
Resource attr filter: style fix and quick filter changes (#7691)
* chore: resource attr filter init
* chore: resource attr filter api integration
* chore: operator config updated
* chore: fliter show hide logic and styles
* chore: add support for custom operator list to qb
* chore: minor refactor
* chore: minor code refactor
* test: quick filters test suite added
* test: quick filters test suite added
* test: all errors test suite added
* chore: style fix
* test: all errors mock fix
* chore: test case fix and mixpanel update
* chore: color update
* chore: minor refactor
* chore: style fix
* chore: set default query in exceptions tab
* chore: style fix
* chore: minor refactor
* chore: minor refactor
* chore: minor refactor
* chore: test update
* chore: fix filter header with no query name
* fix: scroll fix
* chore: add data source traces to quick filters
* chore: replace div with fragment
---------
Co-authored-by: Aditya Singh <adityasingh@Adityas-MacBook-Pro.local>
fix: handle rate operators for table panel (#7695)
* fix: handle rate operators for table panel
chore: fix error rate (#7701)
Signed-off-by: Shivanshu Raj Shrivastava <shivanshu1333@gmail.com>
* feat(organization): minor cleanups
* feat(organization): better naming for api and usecase
* feat(organization): better packaging for modules
* feat(organization): change hname to displayName
* feat(organization): update the migration to use dialect
* feat(organization): update the migration to use dialect
* feat(organization): update the migration to use dialect
* feat(organization): revert back to impl
* feat(organization): remove DI from organization
* feat(organization): address review comments
* feat(organization): address review comments
* feat(organization): address review comments
---------
Signed-off-by: Shivanshu Raj Shrivastava <shivanshu1333@gmail.com>
* chore: multitenancy in integrations
* chore: multitenancy in cloud integration accounts
* chore: changes to cloudintegrationservice
* chore: rename migration
* chore: update scan function
* chore: update scan function
* chore: fix migration
* chore: fix struct
* chore: remove unwanted code
* chore: update scan function
* chore: migrate user and pat for integrations
* fix: changes to the user for integrations
* fix: address comments
* fix: copy created_at
* fix: update non revoked token
* chore: don't allow deleting pat and user for integrations
* fix: address comments
* chore: address comments
* chore: add checks for fk in dialect
* fix: service migration
* fix: don't update user if user is already migrated
* fix: update correct service config
* fix: remove unwanted code
* fix: remove migration for multiple same services which is not required
* fix: fix migration and disable disaboard if metrics disabled
* fix: don't use ee types
---------
Co-authored-by: Vikrant Gupta <vikrant@signoz.io>
* feat(sqlmigration): update the alertmanager tables
* feat(sqlmigration): update the alertmanager tables
* feat(sqlmigration): make the preference package multi tenant
* feat(preference): address nit pick comments
* feat(preference): added the cascade delete for preferences
* feat(sqlmigration): update apdex and TTL status tables (#7481)
* feat(sqlmigration): update the apdex and ttl tables
* feat(sqlmigration): register the new migration and rename table
* feat(sqlmigration): fix the ttl queries
* feat(sqlmigration): update the TTL and apdex tables
* feat(sqlmigration): update the TTL and apdex tables
* fix: support multitenancy in dashboards
* fix: support multitenancy in saved views
* fix: move migrations to provider file
* fix: remove getUserFromClaims and use new errors
* fix: remove getUserFromClaims and use new errors
* fix: use new errors in dashboards.go
* Update ee/query-service/app/api/dashboard.go
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* fix: minor changes
---------
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* fix: support multitenancy in org
* fix: register and login working now
* fix: changes to migration
* fix: migrations run both on sqlite and postgres
* fix: remove user flags from fe and be
* fix: remove ingestion keys from update
* fix: multitenancy support for apdex settings
* fix: render ts for users correctly
* fix: fix migration to run for new tenants
* fix: clean up migrations
* fix: address comments
* Update pkg/sqlmigration/013_update_organization.go
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* fix: fix build
* fix: force invites with org id
* Update pkg/query-service/auth/auth.go
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* fix: address comments
* fix: address comments
* fix: provier with their own dialect
* fix: update dialects
* fix: remove unwanted change
* Update pkg/query-service/app/http_handler.go
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* fix: different files for types
---------
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* chore: include CWAgent namespace for ec2 metrics
* chore: update aws integration service metrics connection status queried labels
* chore: fix breaking test
* chore: get started on dashboards plumbing for AWS integration services
* feat: cloud integrations: include cloud integrations dashboards in dashboards list
* feat: cloud integrations: get cloud integration dashboard by id
* feat: dashboard url in cloud integrations svc detail
* feat: ec2 overview dashboard
* chore: add ec2 overview dashboard image
* chore: finish up with v0 definitions for EC2 and RDS
* chore: some cleanup
* chore: fix broken test
* chore: add connection url composition
---------
Co-authored-by: Srikanth Chekuri <srikanth.chekuri92@gmail.com>
* chore: cloudintegrations: rename rds def
* feat: cloudintegrations: shape of agent check in response for v0 release
* chore: cloudintegrations: add validation for response expected after agent check in
* chore: cloudintegrations: accumulate teletry collection strategies for enabled services
* chore: cloudintegrations: use map struct to parse from map to struct with json tags
* chore: cloudintegrations: telemetry collection strategy for services
* chore: cloudintegrations: wrap up test for agent check in resp
* chore: some cleanup
* chore: some cleanup
* chore: some minor renaming
* chore: address review comment
* chore: some cleanup
---------
Co-authored-by: Srikanth Chekuri <srikanth.chekuri92@gmail.com>
* fix: inital changes for telemetry store
* fix: add tests and use proper config for conn
* fix: add telemetry store test
* fix: add backward compatibility for old variables and update example conf
* fix: move wrapper to telemetry store
* fix: no need to pass query for settings
* fix: remove redundant config for ch conn
* fix: use clickhouse dsn instead
* fix: update example config
* fix: update backward compatibility code
* fix: use hooks in telemetrystore
* fix: address minor comments
---------
Co-authored-by: Vibhu Pandey <vibhupandey28@gmail.com>
* feat: cloud service integrations: get model and repo interface started
* feat: cloud service integrations: flesh out more of cloud services model
* feat: cloud integrations: reorganize things a little
* feat: cloud integrations: get svc controller started
* feat: cloud integrations: add stubs for EC2 and RDS postgres services
* feat: cloud integrations: add validation for listing and getting available svcs and some cleanup
* feat: cloud integrations: refactor helpers in existing integrations code for reuse
* feat: cloud integrations: parsing of cloud service definitions
* feat: cloud integrations: impl for getCloudProviderService
* feat: cloud integrations: some reorganization
* feat: cloud integrations: some more cleanup
* feat: cloud integrations: add validation for listing available cloud provider services
* feat: cloud integrations: API endpoint for listing available cloud provider services
* feat: cloud integrations: add validation for getting details of a particular service
* feat: cloud integrations: API endpoint for getting details of a service
* feat: cloud integrations: add controller validation for configuring cloud services
* feat: cloud integrations: get serviceConfigRepo started
* feat: cloud integrations: service config in service list summaries when queried for cloud account id
* feat: cloud integrations: only a supported service for a connected cloud account can be configured
* feat: cloud integrations: add validation for configuring services via the API
* feat: cloud integrations: API for configuring services
* feat: cloud integrations: some cleanup
* feat: cloud integrations: fix broken test
---------
Co-authored-by: Srikanth Chekuri <srikanth.chekuri92@gmail.com>
* feat: init app/cloud_integrations
* feat: get API test started for cloudintegrations account lifecycle
* feat: cloudintegrations: get controller started
* feat: cloud integrations: add cloudintegrations.Controller to APIHandler and servers
* feat: cloud integrations: get routes started
* feat: cloud integrations: get accounts table schema started
* feat: cloud integrations: get cloudProviderAccountsSQLRepository started
* feat: cloud integrations: cloudProviderAccountsSQLRepository.listAccounts
* feat: cloud integrations: http handler and controller plumbing for /generate-connection-url
* feat: cloud integrations: cloudProviderAccountsSQLRepository.upsert
* feat: cloud integrations: finish up with /generate-connection-url
* feat: cloud integrations: add cloudProviderAccountsRepository.get
* feat: cloud integrations: add API test expectation for being able to get account status
* feat: cloud integrations: add http handler and controller method for getting account status
* feat: cloud integrations: ensure unconnected accounts aren't included in list of connected accounts
* feat: cloud integrations: add test expectation for agent check in request
* feat: cloud integrations: agent check in API
* feat: cloud integrations: ensure polling for status after agent check in works
* feat: cloud integrations: ensure account included in connected account list after agent check in
* feat: cloud integrations: add API expectation for updating account config
* feat: cloud integrations: API for updating cloud account config
* feat: cloud integrations: expectation for agent receiving latest config after account config update
* feat: cloud integrations: expectation for disconnecting cloud accounts from UI
* feat: cloud integrations: API for disconnecting cloud accounts
* feat: cloud integrations: some cleanup
* feat: cloud integrations: some more cleanup
* feat: cloud integrations: repo: scope rows by cloud provider
* feat: testutils: refactor out helper for creating a test sqlite DB
* feat: cloud integrations: controller: add test validating regeneration of connection url
* feat: cloud integrations: controller: validations for agent check ins
* feat: cloud integrations: connected account response structure
* feat: cloud integrations: API response account structure
* feat: cloud integrations: some more cleanup
* feat: cloud integrations: remove cloudProviderAccountsRepository.GetById
* feat: cloud integrations: shouldn't be able to disconnect non-existent account
* feat: cloud integrations: validate agents can't check in to cloud account with 2 signoz ids
* feat: cloud integrations: ensure agents can't check in to cloud account with 2 signoz ids
* feat: cloud integrations: remove stray import of ee/model in cloudintegrations controller
* chore: add test validating invalid field paths in pipeline operators are rejected
* chore: refactor posted pipelines validation to use a controller method
* fix: run a collector simulation to validate pipeline config being saved
* chore: minor cleanup
* feat: trace v4 inital commit
* fix: add remaining files
* fix: integrate with querier
* fix: get trace by id api updated
* fix: add servicename resource filter
* fix: tests
* fix: use correct prepQUery
* fix: services page
* fix: minor fixes to use the new table in api's and querier
* fix: add support for window based pagination
* feat: support for faster trace detail
* fix: searchTraces
* fix: attribute enrichment updated and issue in group by
* fix: issues in group by
* fix: enrichment using alias
* fix: test file added
* fix: tests
* fix: group by with filters
* fix: add subquery
* fix: trigger builde
* fix: update pagination logic and few ch column names
* fix: update qb
* fix: add tests
* feat: minor fixes
* fix: update pagination logic
* fix: update pagination logic
* fix: remove utils
* fix: remove unwanted API's
* fix: attribute and attribute values v2
* fix: autocomplete api's updated
* fix: tests fixed
* feat: minor fixes
* fix: update telemetry functions
* fix: dont use alias, use proper col names
* fix: move models to it's own file
* fix: minor fixes
* fix: address comments
* fix: add to serviceoverview function
* fix: add changes to overview function
* fix: address comments
* fix: remove printlines
---------
Co-authored-by: Srikanth Chekuri <srikanth.chekuri92@gmail.com>
* chore: upgrade signoz-otel-collector dependency to v0.102.10
* feat: first stab at ranking resource attribs higher
* chore: add test todo for validating resource attribs get ranked higher in logs filter suggestions
* chore: add test validating higher ranking for special resource attribs
* chore: some cleanup
* chore: some more cleanup
* chore: put together helper for fetching values for multiple attributes
* chore: poc: use helper for filter suggestions
* chore: add a working impl for getting attrib values for multiple attributes
* chore: start updating integration test to account for new approach for getting log attrib values
* chore: use a global zap logger in filter suggestion tests
* chore: fix attrib values clickhouse query expectation
* chore: only query values for actual attributes when generating example queries
* chore: update clickhouse-go-mock
* chore: cleanup: separate params for attributesLimit and examplesLimit for filter suggestions
* chore: some test cleanup
* chore: some more cleanup
* chore: some more cleanup
---------
Co-authored-by: Srikanth Chekuri <srikanth.chekuri92@gmail.com>
* chore: stash initial work with API signature
* chore: put together setup for integration testing filter suggestions
* feat: filter suggestions: suggest attribs using existing autocomplete logic
* chore: filter suggestions test: add expectation for example queries
* feat: filter suggestions: default suggestions when data yet to be received
* feat: finish plumbing basic example queries
* chore: add test for filter suggestions with an existing query
* feat: filter suggestions: don't suggest attribs already included in existing filter
* chore: generate example queries by including existing filter first
* chore: upgrade ClickHouse-go-mock
* chore: some cleanup of reader.GetQBFilterSuggestionsForLogs
* chore: some cleanup of filter suggestion tests
* chore: some cleanup to http handler and request parsing logic for filter suggestions
* chore: remove expectation that attrib suggestions won't contain attribs already used in filter
