External/buzz
1
0
Fork 0
mirror of https://github.com/block/buzz.git synced 2026-08-18 06:50:31 +02:00
Code Issues Packages Projects Releases Wiki Activity

feat: configure S3 URL addressing style (#3400)

Browse Source 
## Summary

- add one strict `BUZZ_S3_ADDRESSING_STYLE=path|virtual` setting shared
by media and Git/CAS storage
- preserve path-style defaults for bundled Compose/Helm MinIO while
supporting Railway's virtual-hosted bucket contract
- fail startup on invalid or non-Unicode values before dependency
connection, and validate the Helm value with the same two choices
- document operator mappings and why endpoint and bucket remain separate
for routing and SigV4 signing

## Best-practice rationale

AWS documents both URL forms and favors virtual-hosted addressing for
S3, while compatibility endpoints such as the bundled MinIO deployment
can require path style. `rust-s3` defaults to virtual/subdomain
addressing and provides `with_path_style()` for the explicit
compatibility case.

Some providers buckets only support as virtual-hosted bucket styles.
This PR therefore uses one explicit, provider-neutral switch rather than
endpoint heuristics or fallback behavior, while retaining `path` as
Buzz's backward-compatible default.

Sources:
-
https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html
- https://docs.rs/rust-s3/0.37.0/s3/bucket/struct.Bucket.html
- https://docs.railway.com/storage-buckets#url-style
-
https://github.com/minio/minio/blob/master/docs/config/README.md#domain

## Validation

- `cargo fmt --all`
- `cargo check --workspace --all-targets`
- targeted `buzz-media` and `buzz-relay` parsing/client-construction
tests for defaults, strict errors, and both URL styles
- Helm unittest: 45/45 passed
- Compose config/render validation passed
- local MinIO path-mode relay startup passed the Git A3 conformance
probe and became ready
- unreachable object storage failed startup and readiness never opened
- push hooks completed the broader Rust and desktop suites successfully

---------

Signed-off-by: npub122y0pqkertljmedu303rl0aqrj3w8pvu43t6jxm6875lzg6f2pwqegc3xc <5288f082d91aff2de5bc8be23fbfa01ca2e3859cac57a91b7a3fa9f12349505c@buzz.block.builderlab.xyz>
Signed-off-by: Kalvin Chau <kalvin@block.xyz>
Co-authored-by: npub122y0pqkertljmedu303rl0aqrj3w8pvu43t6jxm6875lzg6f2pwqegc3xc <5288f082d91aff2de5bc8be23fbfa01ca2e3859cac57a91b7a3fa9f12349505c@buzz.block.builderlab.xyz>
This commit is contained in:
Kalvin C
2026-07-29 17:00:41 -07:00
committed by GitHub
co-authored by npub122y0pqkertljmedu303rl0aqrj3w8pvu43t6jxm6875lzg6f2pwqegc3xc
parent ab55fee818
commit 7012d86d52
28 changed files with 471 additions and 60 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.env.example
+13
View File
@@ -82,6 +82,19 @@ RELAY_URL=ws://localhost:3000
# BUZZ_GIT_PACK_CACHE_MAX_BYTES=5368709120
# BUZZ_GIT_PACK_CACHE_MAX_CONCURRENT_POPULATIONS=2
# -----------------------------------------------------------------------------
# S3-Compatible Object Storage (media + Git/CAS)
# -----------------------------------------------------------------------------
# The local MinIO container is reachable from host processes at localhost:9000.
# Path style keeps the bucket in the URL path and is required by this local DNS
# setup. Use `virtual` only when the provider requires bucket-as-subdomain URLs.
BUZZ_S3_ENDPOINT=http://localhost:9000
BUZZ_S3_ACCESS_KEY=buzz_dev
BUZZ_S3_SECRET_KEY=buzz_dev_secret
BUZZ_S3_BUCKET=buzz-media
BUZZ_S3_REGION=us-east-1
BUZZ_S3_ADDRESSING_STYLE=path
# -----------------------------------------------------------------------------
# Media Upload Admission
# -----------------------------------------------------------------------------
Cargo.lock
Generated
+1
View File
@@ -1239,6 +1239,7 @@ dependencies = [
"anyhow",
"base64",
"buzz-core",
"buzz-media",
"buzz-sdk",
"buzz-ws-client",
"chrono",
crates/buzz-media/src/config.rs
+68 -1
View File
@@ -1,5 +1,38 @@
//! Media storage configuration.
use std::str::FromStr;
/// S3 URL addressing style shared by media and Git/CAS storage.
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, serde::Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum S3AddressingStyle {
/// Put the bucket in the request path (`https://endpoint/bucket/key`).
///
/// This preserves compatibility with the bundled MinIO deployments, whose
/// internal DNS only resolves the endpoint hostname.
#[default]
Path,
/// Put the bucket in the hostname (`https://bucket.endpoint/key`).
///
/// This is the standard S3 form and is required by providers such as new
/// Railway Storage Buckets.
Virtual,
}
impl FromStr for S3AddressingStyle {
type Err = String;
fn from_str(value: &str) -> Result<Self, Self::Err> {
match value {
"path" => Ok(Self::Path),
"virtual" => Ok(Self::Virtual),
_ => Err(format!(
"BUZZ_S3_ADDRESSING_STYLE must be 'path' or 'virtual', got {value:?}"
)),
}
}
}
fn default_max_video_bytes() -> u64 {
524_288_000 // 500 MB
}
@@ -31,6 +64,9 @@ pub struct MediaConfig {
/// the value is not meaningfully checked.
#[serde(default = "default_s3_region")]
pub s3_region: String,
/// S3 URL addressing style. Defaults to path style for MinIO compatibility.
#[serde(default)]
pub s3_addressing_style: S3AddressingStyle,
/// Maximum upload size for images (bytes). Default: 50 MB.
pub max_image_bytes: u64,
/// Maximum upload size for animated GIFs (bytes). Default: 10 MB.
@@ -123,7 +159,8 @@ impl MediaConfig {
#[cfg(test)]
mod tests {
use super::MediaConfig;
use super::{MediaConfig, S3AddressingStyle};
use std::str::FromStr;
fn valid_config() -> MediaConfig {
MediaConfig {
@@ -132,6 +169,7 @@ mod tests {
s3_secret_key: "s".to_string(),
s3_bucket: "buzz-media".to_string(),
s3_region: "us-east-1".to_string(),
s3_addressing_style: S3AddressingStyle::Path,
max_image_bytes: 1,
max_gif_bytes: 1,
max_video_bytes: 1,
@@ -143,6 +181,35 @@ mod tests {
}
}
#[test]
fn addressing_style_parses_supported_values() {
assert_eq!(
S3AddressingStyle::from_str("path"),
Ok(S3AddressingStyle::Path)
);
assert_eq!(
S3AddressingStyle::from_str("virtual"),
Ok(S3AddressingStyle::Virtual)
);
}
#[test]
fn addressing_style_defaults_to_path() {
assert_eq!(S3AddressingStyle::default(), S3AddressingStyle::Path);
}
#[test]
fn addressing_style_rejects_unknown_or_ambiguous_values() {
for invalid in ["", "auto", "PATH", "virtual-hosted"] {
let error =
S3AddressingStyle::from_str(invalid).expect_err("must reject invalid style");
assert!(
error.contains("BUZZ_S3_ADDRESSING_STYLE must be 'path' or 'virtual'"),
"unexpected error for {invalid:?}: {error}"
);
}
}
#[test]
fn upload_record_knobs_default_off_and_validate() {
assert!(valid_config().validate().is_ok());
crates/buzz-media/src/lib.rs
+1 -1
View File
@@ -17,7 +17,7 @@ pub use bucket_index::{
classify_key, fold_bucket_listing, BucketAggregate, BucketSnapshot, CommunityStorage, KeyClass,
Page, SweepError,
};
pub use config::MediaConfig;
pub use config::{MediaConfig, S3AddressingStyle};
pub use error::MediaError;
pub use storage::{BlobHeadMeta, BlobMeta, ByteStream, MediaStorage};
pub use types::BlobDescriptor;
crates/buzz-media/src/storage.rs
+24 -3
View File
@@ -5,7 +5,7 @@ use std::pin::Pin;
use buzz_core::tenant::{CommunityId, TenantContext};
use crate::config::MediaConfig;
use crate::config::{MediaConfig, S3AddressingStyle};
use crate::error::MediaError;
use bytes::Bytes;
use s3::creds::Credentials;
@@ -61,8 +61,11 @@ impl MediaStorage {
}
.map_err(|e| MediaError::StorageError(e.to_string()))?;
let bucket = Bucket::new(&config.s3_bucket, region, creds)
.map_err(|e| MediaError::StorageError(e.to_string()))?
.with_path_style();
.map_err(|e| MediaError::StorageError(e.to_string()))?;
let bucket = match config.s3_addressing_style {
S3AddressingStyle::Path => bucket.with_path_style(),
S3AddressingStyle::Virtual => bucket,
};
Ok(Self { bucket })
}
@@ -285,6 +288,7 @@ mod tests {
s3_secret_key: secret.to_string(),
s3_bucket: "buzz-media".to_string(),
s3_region: "us-west-2".to_string(),
s3_addressing_style: S3AddressingStyle::Path,
max_image_bytes: 50 * 1024 * 1024,
max_gif_bytes: 10 * 1024 * 1024,
max_video_bytes: 524_288_000,
@@ -309,6 +313,23 @@ mod tests {
}
}
#[test]
fn client_constructor_applies_both_addressing_styles() {
let path = MediaStorage::new(&storage_config("buzz_dev", "buzz_dev_secret"))
.expect("path-style client");
assert!(path.bucket.is_path_style());
assert_eq!(path.bucket.url(), "http://localhost:9000/buzz-media");
let mut virtual_config = storage_config("buzz_dev", "buzz_dev_secret");
virtual_config.s3_addressing_style = S3AddressingStyle::Virtual;
let virtual_hosted = MediaStorage::new(&virtual_config).expect("virtual-hosted client");
assert!(virtual_hosted.bucket.is_subdomain_style());
assert_eq!(
virtual_hosted.bucket.url(),
"http://buzz-media.localhost:9000"
);
}
#[test]
fn partial_static_keys_are_rejected() {
let err = match MediaStorage::new(&storage_config("buzz_dev", "")) {
crates/buzz-media/src/upload.rs
+1
View File
@@ -570,6 +570,7 @@ mod tests {
s3_secret_key: String::new(),
s3_bucket: String::new(),
s3_region: "us-east-1".to_string(),
s3_addressing_style: crate::config::S3AddressingStyle::Path,
max_image_bytes: 50 * 1024 * 1024,
max_gif_bytes: 10 * 1024 * 1024,
max_video_bytes: 524_288_000,
crates/buzz-media/src/validation.rs
+1
View File
@@ -949,6 +949,7 @@ mod tests {
s3_secret_key: String::new(),
s3_bucket: String::new(),
s3_region: "us-east-1".to_string(),
s3_addressing_style: crate::config::S3AddressingStyle::Path,
max_image_bytes: 50 * 1024 * 1024,
max_gif_bytes: 10 * 1024 * 1024,
max_video_bytes: 524_288_000,
crates/buzz-media/tests/static_creds_minio.rs
+9 -4
View File
@@ -1,5 +1,5 @@
//! Live round-trip test for the **static-credentials** S3 path against a local
//! MinIO, guarded by `#[ignore]`.
//! Live round-trip test for the **static-credentials** S3 path against an
//! S3-compatible service. It is guarded by `#[ignore]`.
//!
//! This is the path local/dev and any static-key deployment uses
//! (`s3_access_key`/`s3_secret_key` both non-empty -> `Credentials::new`). It
@@ -15,7 +15,8 @@
//! ```
//!
//! Overridable via `BUZZ_S3_ENDPOINT` / `BUZZ_S3_ACCESS_KEY` /
//! `BUZZ_S3_SECRET_KEY` / `BUZZ_S3_BUCKET`.
//! `BUZZ_S3_SECRET_KEY` / `BUZZ_S3_BUCKET` / `BUZZ_S3_REGION` /
//! `BUZZ_S3_ADDRESSING_STYLE`. The default remains `path` for MinIO.
use buzz_media::config::MediaConfig;
use buzz_media::storage::MediaStorage;
@@ -29,7 +30,11 @@ fn minio_config() -> MediaConfig {
s3_secret_key: std::env::var("BUZZ_S3_SECRET_KEY")
.unwrap_or_else(|_| "buzz_dev_secret".to_string()),
s3_bucket: std::env::var("BUZZ_S3_BUCKET").unwrap_or_else(|_| "buzz-media".to_string()),
s3_region: "us-east-1".to_string(),
s3_region: std::env::var("BUZZ_S3_REGION").unwrap_or_else(|_| "us-east-1".to_string()),
s3_addressing_style: std::env::var("BUZZ_S3_ADDRESSING_STYLE")
.unwrap_or_else(|_| "path".to_string())
.parse()
.expect("BUZZ_S3_ADDRESSING_STYLE must be path or virtual"),
max_image_bytes: 50 * 1024 * 1024,
max_gif_bytes: 10 * 1024 * 1024,
max_video_bytes: 524_288_000,
crates/buzz-relay/src/api/git/cas_publish.rs
+9 -16
View File
@@ -1583,22 +1583,15 @@ mod tests {
}
fn live_store() -> GitStore {
let endpoint = std::env::var("BUZZ_GIT_S3_ENDPOINT")
.or_else(|_| std::env::var("BUZZ_S3_ENDPOINT"))
.unwrap_or_else(|_| "http://localhost:9000".into());
let access_key = std::env::var("BUZZ_GIT_S3_ACCESS_KEY")
.or_else(|_| std::env::var("BUZZ_S3_ACCESS_KEY"))
.unwrap_or_else(|_| "buzz_dev".into());
let secret_key = std::env::var("BUZZ_GIT_S3_SECRET_KEY")
.or_else(|_| std::env::var("BUZZ_S3_SECRET_KEY"))
.unwrap_or_else(|_| "buzz_dev_secret".into());
let bucket = std::env::var("BUZZ_GIT_S3_BUCKET")
.or_else(|_| std::env::var("BUZZ_S3_BUCKET"))
.unwrap_or_else(|_| "buzz-media".into());
let region = std::env::var("BUZZ_GIT_S3_REGION")
.or_else(|_| std::env::var("BUZZ_S3_REGION"))
.unwrap_or_else(|_| "us-east-1".into());
GitStore::new(&endpoint, &access_key, &secret_key, &bucket, &region).expect("connect minio")
GitStore::new(
"http://localhost:9000",
"buzz_dev",
"buzz_dev_secret",
"buzz-media",
"us-east-1",
buzz_media::config::S3AddressingStyle::Path,
)
.expect("connect local MinIO")
}
fn tenant() -> TenantContext {
crates/buzz-relay/src/api/git/hydrate.rs
+11 -3
View File
@@ -543,8 +543,15 @@ mod tests {
#[tokio::test]
async fn materialized_repo_is_created_under_configured_scratch_dir() {
let scratch = TempDir::new().unwrap();
let store = GitStore::new("http://localhost:9000", "x", "x", "x", "us-east-1")
.expect("construct store");
let store = GitStore::new(
"http://localhost:9000",
"x",
"x",
"x",
"us-east-1",
buzz_media::config::S3AddressingStyle::Path,
)
.expect("construct store");
let manifest = Manifest {
version: 1,
head: "refs/heads/main".into(),
@@ -587,8 +594,9 @@ mod tests {
"buzz_dev_secret",
"buzz-git",
"us-east-1",
buzz_media::config::S3AddressingStyle::Path,
)
.expect("connect minio")
.expect("connect local MinIO")
}
/// Build a tiny on-disk repo, return (pack bytes, head_oid).
crates/buzz-relay/src/api/git/store.rs
+60 -10
View File
@@ -174,7 +174,9 @@ pub struct GitStore {
impl GitStore {
/// Build a client against an S3-compatible endpoint (e.g. MinIO).
///
/// Uses path-style addressing for MinIO compatibility; AWS S3 accepts both.
/// `addressing_style` is shared with media storage so both paths sign and
/// route requests consistently. Path style supports the bundled MinIO DNS;
/// virtual-hosted style supports standard S3 and providers such as Railway.
///
/// Credential selection mirrors [`buzz_media::MediaStorage::new`]:
/// - both `access_key` and `secret_key` non-empty → static credentials
@@ -190,6 +192,7 @@ impl GitStore {
secret_key: &str,
bucket_name: &str,
region: &str,
addressing_style: buzz_media::config::S3AddressingStyle,
) -> Result<Self, StoreError> {
let region = Region::Custom {
region: region.into(),
@@ -209,9 +212,11 @@ impl GitStore {
}
}
.map_err(|e| StoreError::Backend(S3Error::Credentials(e)))?;
let bucket = Bucket::new(bucket_name, region, creds)
.map_err(StoreError::Backend)?
.with_path_style();
let bucket = Bucket::new(bucket_name, region, creds).map_err(StoreError::Backend)?;
let bucket = match addressing_style {
buzz_media::config::S3AddressingStyle::Path => bucket.with_path_style(),
buzz_media::config::S3AddressingStyle::Virtual => bucket,
};
Ok(Self {
bucket: Arc::from(bucket),
})
@@ -950,6 +955,7 @@ mod tests {
"buzz_dev_secret",
"buzz-git",
"us-west-2",
buzz_media::config::S3AddressingStyle::Path,
)
.expect("static creds should build a git store");
match store.bucket.region {
@@ -958,6 +964,34 @@ mod tests {
}
}
#[test]
fn constructor_applies_both_addressing_styles() {
for (style, expected_url, path_style) in [
(
buzz_media::config::S3AddressingStyle::Path,
"https://storage.example/buzz-git",
true,
),
(
buzz_media::config::S3AddressingStyle::Virtual,
"https://buzz-git.storage.example",
false,
),
] {
let store = GitStore::new(
"https://storage.example",
"buzz_dev",
"buzz_dev_secret",
"buzz-git",
"us-east-1",
style,
)
.expect("construct git store");
assert_eq!(store.bucket.url(), expected_url);
assert_eq!(store.bucket.is_path_style(), path_style);
}
}
#[test]
fn partial_static_keys_are_rejected() {
for (access, secret) in [("buzz_dev", ""), ("", "buzz_dev_secret")] {
@@ -967,6 +1001,7 @@ mod tests {
secret,
"buzz-git",
"us-east-1",
buzz_media::config::S3AddressingStyle::Path,
) {
Ok(_) => {
panic!("partial static creds must not silently use the credential chain")
@@ -998,14 +1033,29 @@ mod probe {
}
fn store() -> GitStore {
// This is the dedicated backend conformance path, so all connection and
// signing inputs are overridable for a real provider such as Railway.
// The hydrate/CAS live tests use explicit local MinIO fixtures instead.
let endpoint =
std::env::var("BUZZ_S3_ENDPOINT").unwrap_or_else(|_| "http://localhost:9000".into());
let access_key = std::env::var("BUZZ_S3_ACCESS_KEY").unwrap_or_else(|_| "buzz_dev".into());
let secret_key =
std::env::var("BUZZ_S3_SECRET_KEY").unwrap_or_else(|_| "buzz_dev_secret".into());
let bucket = std::env::var("BUZZ_S3_BUCKET").unwrap_or_else(|_| "buzz-git".into());
let region = std::env::var("BUZZ_S3_REGION").unwrap_or_else(|_| "us-east-1".into());
let addressing_style = std::env::var("BUZZ_S3_ADDRESSING_STYLE")
.unwrap_or_else(|_| "path".into())
.parse()
.expect("BUZZ_S3_ADDRESSING_STYLE must be path or virtual");
GitStore::new(
"http://localhost:9000",
"buzz_dev",
"buzz_dev_secret",
"buzz-git",
"us-east-1",
&endpoint,
&access_key,
&secret_key,
&bucket,
&region,
addressing_style,
)
.expect("connect minio")
.expect("connect S3-compatible storage")
}
fn sha256_hex(b: &[u8]) -> String {
crates/buzz-relay/src/config.rs
+71
View File
@@ -630,6 +630,16 @@ impl Config {
.and_then(|v| v.parse().ok())
.unwrap_or(9102);
let s3_addressing_style = match std::env::var("BUZZ_S3_ADDRESSING_STYLE") {
Ok(value) => value.parse().map_err(ConfigError::InvalidValue)?,
Err(std::env::VarError::NotPresent) => buzz_media::config::S3AddressingStyle::default(),
Err(std::env::VarError::NotUnicode(_)) => {
return Err(ConfigError::InvalidValue(
"BUZZ_S3_ADDRESSING_STYLE must be valid Unicode and one of 'path' or 'virtual'"
.to_string(),
));
}
};
let media = buzz_media::MediaConfig {
s3_endpoint: std::env::var("BUZZ_S3_ENDPOINT")
.unwrap_or_else(|_| "http://localhost:9000".to_string()),
@@ -641,6 +651,7 @@ impl Config {
s3_region: std::env::var("BUZZ_S3_REGION")
.or_else(|_| std::env::var("AWS_REGION"))
.unwrap_or_else(|_| "us-east-1".to_string()),
s3_addressing_style,
max_image_bytes: std::env::var("BUZZ_MAX_IMAGE_BYTES")
.ok()
.and_then(|v| v.parse().ok())
@@ -990,6 +1001,11 @@ mod tests {
!config.require_media_get_auth,
"require_media_get_auth should default to false for staged client rollout"
);
assert_eq!(
config.media.s3_addressing_style,
buzz_media::config::S3AddressingStyle::Path,
"S3 addressing must default to path style for bundled MinIO compatibility"
);
assert!(
config.join_policy.is_none(),
"join_policy should default to None so policy prompts and acceptance receipts are opt-in"
@@ -1000,6 +1016,61 @@ mod tests {
);
}
#[test]
fn s3_addressing_style_env_accepts_virtual_and_rejects_invalid_values() {
let _guard = ENV_MUTEX.lock().unwrap();
let previous = std::env::var_os("BUZZ_S3_ADDRESSING_STYLE");
std::env::set_var("BUZZ_S3_ADDRESSING_STYLE", "virtual");
let configured = Config::from_env()
.expect("virtual style config")
.media
.s3_addressing_style;
std::env::set_var("BUZZ_S3_ADDRESSING_STYLE", "auto");
let invalid = Config::from_env();
if let Some(value) = previous {
std::env::set_var("BUZZ_S3_ADDRESSING_STYLE", value);
} else {
std::env::remove_var("BUZZ_S3_ADDRESSING_STYLE");
}
assert_eq!(configured, buzz_media::config::S3AddressingStyle::Virtual);
assert!(matches!(
invalid,
Err(ConfigError::InvalidValue(ref message))
if message.contains("BUZZ_S3_ADDRESSING_STYLE must be 'path' or 'virtual'")
));
}
#[cfg(unix)]
#[test]
fn s3_addressing_style_env_rejects_non_unicode_values() {
use std::os::unix::ffi::OsStringExt;
let _guard = ENV_MUTEX.lock().unwrap();
let previous = std::env::var_os("BUZZ_S3_ADDRESSING_STYLE");
std::env::set_var(
"BUZZ_S3_ADDRESSING_STYLE",
std::ffi::OsString::from_vec(vec![0xff]),
);
let invalid = Config::from_env();
if let Some(value) = previous {
std::env::set_var("BUZZ_S3_ADDRESSING_STYLE", value);
} else {
std::env::remove_var("BUZZ_S3_ADDRESSING_STYLE");
}
assert!(matches!(
invalid,
Err(ConfigError::InvalidValue(ref message))
if message.contains("must be valid Unicode")
));
}
#[test]
fn redis_pool_size_env_override_and_invalid_fallback() {
let _guard = ENV_MUTEX.lock().unwrap();
crates/buzz-relay/src/state.rs
+1
View File
@@ -697,6 +697,7 @@ impl AppState {
&config.media.s3_secret_key,
&config.media.s3_bucket,
&config.media.s3_region,
config.media.s3_addressing_style,
)
.expect("media storage was already constructed with this S3 config");
let git_pack_cache = Arc::new(
crates/buzz-test-client/Cargo.toml
+1
View File
@@ -36,6 +36,7 @@ sha2 = { workspace = true }
sqlx = { workspace = true }
chrono = { workspace = true }
s3 = { version = "0.37", package = "rust-s3", default-features = false, features = ["tokio-rustls-tls", "fail-on-err", "tags"] }
buzz-media = { workspace = true }
buzz-sdk = { workspace = true }
[[bin]]
crates/buzz-test-client/tests/e2e_git.rs
+71 -19
View File
@@ -21,6 +21,7 @@ use std::path::{Path, PathBuf};
use std::process::Command;
use std::time::Duration;
use buzz_media::S3AddressingStyle;
use nostr::{EventBuilder, Keys, Kind, Tag};
use s3::creds::Credentials;
use s3::{Bucket, Region};
@@ -115,29 +116,55 @@ struct PointerSnapshot {
}
impl GitS3Probe {
fn from_env() -> Self {
let endpoint = std::env::var("BUZZ_GIT_S3_ENDPOINT")
.or_else(|_| std::env::var("BUZZ_S3_ENDPOINT"))
.unwrap_or_else(|_| "http://localhost:9000".to_string());
let access_key = std::env::var("BUZZ_GIT_S3_ACCESS_KEY")
.or_else(|_| std::env::var("BUZZ_S3_ACCESS_KEY"))
.unwrap_or_else(|_| "buzz_dev".to_string());
let secret_key = std::env::var("BUZZ_GIT_S3_SECRET_KEY")
.or_else(|_| std::env::var("BUZZ_S3_SECRET_KEY"))
.unwrap_or_else(|_| "buzz_dev_secret".to_string());
let bucket = std::env::var("BUZZ_GIT_S3_BUCKET")
.or_else(|_| std::env::var("BUZZ_S3_BUCKET"))
.unwrap_or_else(|_| "buzz-media".to_string());
fn bucket(
endpoint: String,
access_key: &str,
secret_key: &str,
bucket_name: &str,
region_name: String,
addressing_style: S3AddressingStyle,
) -> Box<Bucket> {
let region = Region::Custom {
region: "us-east-1".into(),
region: region_name,
endpoint,
};
let creds = Credentials::new(Some(&access_key), Some(&secret_key), None, None, None)
let creds = Credentials::new(Some(access_key), Some(secret_key), None, None, None)
.expect("S3 credentials");
let bucket = Bucket::new(&bucket, region, creds)
.expect("S3 bucket")
.with_path_style();
let bucket = Bucket::new(bucket_name, region, creds).expect("S3 bucket");
match addressing_style {
S3AddressingStyle::Path => bucket.with_path_style(),
S3AddressingStyle::Virtual => bucket,
}
}
fn from_env() -> Self {
// These E2E assertions inspect the relay's backing bucket directly, so
// they must receive the same provider connection and URL style as the
// relay. Unit/live MinIO probes in buzz-relay keep explicit local
// fixtures and do not need provider overrides.
let endpoint = std::env::var("BUZZ_S3_ENDPOINT")
.unwrap_or_else(|_| "http://localhost:9000".to_string());
let access_key =
std::env::var("BUZZ_S3_ACCESS_KEY").unwrap_or_else(|_| "buzz_dev".to_string());
let secret_key =
std::env::var("BUZZ_S3_SECRET_KEY").unwrap_or_else(|_| "buzz_dev_secret".to_string());
let bucket_name =
std::env::var("BUZZ_S3_BUCKET").unwrap_or_else(|_| "buzz-media".to_string());
let region_name =
std::env::var("BUZZ_S3_REGION").unwrap_or_else(|_| "us-east-1".to_string());
let addressing_style = std::env::var("BUZZ_S3_ADDRESSING_STYLE")
.unwrap_or_else(|_| "path".to_string())
.parse::<S3AddressingStyle>()
.expect("BUZZ_S3_ADDRESSING_STYLE must be 'path' or 'virtual'");
let bucket = Self::bucket(
endpoint,
&access_key,
&secret_key,
&bucket_name,
region_name,
addressing_style,
);
Self { bucket }
}
@@ -192,6 +219,31 @@ impl GitS3Probe {
}
}
#[test]
fn git_s3_probe_builds_both_addressing_styles() {
let path = GitS3Probe::bucket(
"https://storage.example".to_string(),
"access",
"secret",
"buzz-media",
"us-east-1".to_string(),
S3AddressingStyle::Path,
);
assert!(path.is_path_style());
assert_eq!(path.url(), "https://storage.example/buzz-media");
let virtual_hosted = GitS3Probe::bucket(
"https://storage.example".to_string(),
"access",
"secret",
"buzz-media",
"auto".to_string(),
S3AddressingStyle::Virtual,
);
assert!(virtual_hosted.is_subdomain_style());
assert_eq!(virtual_hosted.url(), "https://buzz-media.storage.example");
}
#[tokio::test]
#[ignore = "requires live relay + MinIO + git"]
async fn git_clone_push_fetch_force_roundtrip() {
deploy/charts/buzz/README.md
+42
View File
@@ -52,6 +52,48 @@ See:
The chart fails at `helm install` / `helm template` time with a clear message if any of these are missing or malformed (see `templates/_validate.tpl`).
## S3 URL addressing
Buzz uses one URL style for both media and Git/CAS object-store requests:
| `s3.addressingStyle` | Request shape | Use for |
|---|---|---|
| `path` (default) | `https://endpoint/bucket/key` | Bundled MinIO and endpoints whose DNS does not resolve bucket subdomains |
| `virtual` | `https://bucket.endpoint/key` | AWS-style providers and new Railway Storage Buckets |
The chart always renders `s3.addressingStyle` as
`BUZZ_S3_ADDRESSING_STYLE`. It renders `s3.region` as `BUZZ_S3_REGION` only
when explicitly set, preserving the relay's existing `AWS_REGION` fallback for
upgrades. Only `path` and `virtual` addressing styles are accepted; invalid
values fail chart rendering and relay startup. The bundled MinIO quickstart
deliberately keeps `path` because its Service DNS resolves one endpoint
hostname, not arbitrary `<bucket>.<service>` names.
For a Railway Storage Bucket, map its variables to chart values in the service
or generated Helm configuration:
```yaml
s3:
endpoint: "${{Object Storage.ENDPOINT}}"
bucket: "${{Object Storage.BUCKET}}"
region: "${{Object Storage.REGION}}"
addressingStyle: virtual
```
Store `BUZZ_S3_ACCESS_KEY=${{Object Storage.ACCESS_KEY_ID}}` and
`BUZZ_S3_SECRET_KEY=${{Object Storage.SECRET_ACCESS_KEY}}` in the Secret named by
`secrets.existingSecret`. Railway's Credentials tab is authoritative for older
buckets, which may still require `path`. The setting changes request routing and
SigV4 signing, so do not put the bucket into `s3.endpoint`; pass Railway's base
`ENDPOINT` and `BUCKET` separately.
Object storage is contacted during relay startup only when
`BUZZ_GIT_CONFORMANCE_PROBE` is enabled (the relay default). A probe failure is
startup-fatal, so Kubernetes readiness never opens. If an operator explicitly
disables that probe through `relay.extraEnv`, `/_readiness` does not test object
storage; configuration is still parsed strictly, but reachability and addressing
errors surface on the first storage operation.
## Relay Pod extensions
The chart exposes narrow extension points for init containers, volumes, relay
deploy/charts/buzz/examples/argocd-app.yaml
+2
View File
@@ -41,6 +41,8 @@ spec:
s3:
endpoint: "https://s3.us-east-1.amazonaws.com"
bucket: "buzz-media"
region: "us-east-1"
addressingStyle: virtual
# accessKey / secretKey live in buzz-secrets
persistence:
deploy/charts/buzz/examples/flux-helmrelease.yaml
+2
View File
@@ -41,6 +41,8 @@ spec:
s3:
endpoint: "https://s3.us-east-1.amazonaws.com"
bucket: "buzz-media"
region: "us-east-1"
addressingStyle: virtual
persistence:
git:
deploy/charts/buzz/templates/_validate.tpl
+5 -3
View File
@@ -75,10 +75,12 @@ surface at template time regardless of which manifest helm renders first.
{{- fail "Postgres source missing: enable postgresql.enabled=true, set externalPostgresql.url, or provide secrets.existingSecret with key DATABASE_URL." -}}
{{- end -}}
{{/* S3 / object-storage source must exist somewhere (relay hard-fails its
startup conformance probe without a reachable bucket). */}}
{{/* S3 / object-storage source must exist somewhere. With the default
BUZZ_GIT_CONFORMANCE_PROBE behavior, an unreachable bucket is detected
before the relay opens its listener; operators can explicitly disable that
startup gate. */}}
{{- if not (or .Values.minio.enabled .Values.s3.endpoint .Values.secrets.existingSecret) -}}
{{- fail "S3/object-storage source missing: enable minio.enabled=true (quickstart in-cluster), set s3.endpoint + s3.bucket + credentials, or provide secrets.existingSecret with keys BUZZ_S3_ACCESS_KEY + BUZZ_S3_SECRET_KEY. The relay runs a startup S3 conformance probe and exits if storage is unreachable." -}}
{{- fail "S3/object-storage source missing: enable minio.enabled=true (quickstart in-cluster), set s3.endpoint + s3.bucket + credentials, or provide secrets.existingSecret with keys BUZZ_S3_ACCESS_KEY + BUZZ_S3_SECRET_KEY. By default the relay runs a startup S3 conformance probe and exits if storage is unreachable; disabling BUZZ_GIT_CONFORMANCE_PROBE also removes that startup storage check." -}}
{{- end -}}
{{- end -}}
deploy/charts/buzz/templates/deployment.yaml
+4
View File
@@ -170,6 +170,10 @@ spec:
- { name: BUZZ_S3_ENDPOINT, value: {{ $s3Endpoint | quote }} }
{{- end }}
- { name: BUZZ_S3_BUCKET, value: {{ .Values.s3.bucket | quote }} }
{{- if .Values.s3.region }}
- { name: BUZZ_S3_REGION, value: {{ .Values.s3.region | quote }} }
{{- end }}
- { name: BUZZ_S3_ADDRESSING_STYLE, value: {{ .Values.s3.addressingStyle | quote }} }
# ── Secrets (from chart-managed or existing) ─────────────
- name: BUZZ_RELAY_PRIVATE_KEY
deploy/charts/buzz/tests/render_test.yaml
+38
View File
@@ -30,6 +30,18 @@ tests:
path: kind
value: Service
template: templates/service.yaml
- notContains:
path: spec.template.spec.containers[0].env
content:
name: BUZZ_S3_REGION
any: true
template: templates/deployment.yaml
- contains:
path: spec.template.spec.containers[0].env
content:
name: BUZZ_S3_ADDRESSING_STYLE
value: "path"
template: templates/deployment.yaml
- contains:
path: spec.template.spec.containers[0].env
content:
@@ -47,6 +59,32 @@ tests:
value: "true"
template: templates/deployment.yaml
- it: renders virtual-hosted S3 addressing for providers that require it
set:
relayUrl: wss://buzz.example.com
ownerPubkey: "0000000000000000000000000000000000000000000000000000000000000000"
externalPostgresql.url: postgres://u:p@h:5432/d
externalRedis.url: redis://h:6379
s3.endpoint: https://storage.railway.app
s3.bucket: buzz-media-example
s3.region: auto
s3.addressingStyle: virtual
s3.accessKey: a
s3.secretKey: s
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: BUZZ_S3_REGION
value: "auto"
template: templates/deployment.yaml
- contains:
path: spec.template.spec.containers[0].env
content:
name: BUZZ_S3_ADDRESSING_STYLE
value: "virtual"
template: templates/deployment.yaml
- it: lets an explicit value opt out of media read auth for dev/public deployments
set:
relayUrl: wss://buzz.example.com
deploy/charts/buzz/tests/validation_test.yaml
+11
View File
@@ -58,6 +58,17 @@ tests:
- failedTemplate:
errorPattern: "Postgres source missing"
- it: rejects an invalid S3 addressing style
set:
relayUrl: wss://buzz.example.com
ownerPubkey: "0000000000000000000000000000000000000000000000000000000000000000"
externalPostgresql.url: postgres://u:p@h:5432/d
s3.endpoint: http://minio:9000
s3.addressingStyle: auto
asserts:
- failedTemplate:
errorPattern: "s3.addressingStyle: s3.addressingStyle must be one of the following:.*path.*virtual"
- it: fails when S3/object-storage source is missing
set:
relayUrl: wss://buzz.example.com
deploy/charts/buzz/values.schema.json
+9
View File
@@ -198,6 +198,15 @@
"properties": {
"endpoint": { "type": "string", "pattern": "^(https?://.+)?$" },
"bucket": { "type": "string", "minLength": 1 },
"region": {
"type": "string",
"description": "Optional S3 region used for SigV4 signing. When empty, BUZZ_S3_REGION is omitted so the relay can use AWS_REGION or its own default."
},
"addressingStyle": {
"type": "string",
"enum": ["path", "virtual"],
"description": "S3 URL style shared by media and Git/CAS clients. Defaults to path for bundled MinIO compatibility."
},
"accessKey": { "type": "string" },
"secretKey": { "type": "string" }
}
deploy/charts/buzz/values.yaml
+6
View File
@@ -338,6 +338,12 @@ externalRedis:
s3:
endpoint: ""
bucket: "buzz-media"
# Optional SigV4 signing region. Leave empty to preserve the relay's
# AWS_REGION fallback; set the provider's credential value when needed.
region: ""
# path: https://endpoint/bucket/key (bundled MinIO-compatible default)
# virtual: https://bucket.endpoint/key (standard S3; required by new Railway buckets)
addressingStyle: path
accessKey: ""
secretKey: ""
deploy/compose/.env.example
+2
View File
@@ -33,6 +33,8 @@ REDIS_PASSWORD=CHANGE_ME_RANDOM_PASSWORD
BUZZ_S3_ACCESS_KEY=CHANGE_ME_RANDOM_ACCESS_KEY
BUZZ_S3_SECRET_KEY=CHANGE_ME_RANDOM_SECRET_KEY
BUZZ_S3_BUCKET=buzz-media
# Bundled MinIO uses path-style URLs; deploy/compose/compose.yml pins this.
BUZZ_S3_ADDRESSING_STYLE=path
# Optional host ports. Base compose publishes the relay directly on BUZZ_HTTP_PORT.
BUZZ_HTTP_PORT=3000
deploy/compose/README.md
+5
View File
@@ -38,6 +38,11 @@ keypair.
migrations.
- The stack uses Postgres, Redis, MinIO, and a git data volume because
those are real Buzz dependencies today. Minimal mode can simplify this later.
- The bundled Compose stack fixes the relay endpoint to `http://minio:9000` and
`BUZZ_S3_ADDRESSING_STYLE=path`: Docker DNS resolves `minio`, not
`<bucket>.minio`. It is not configurable for an external S3 provider through
`.env`; use the Helm chart or a custom Compose configuration for providers
such as new Railway Storage Buckets that require `virtual` addressing.
Run `./run.sh backup-hint` for the backup checklist.
deploy/compose/compose.yml
+2
View File
@@ -12,6 +12,8 @@ services:
DATABASE_URL: postgres://${POSTGRES_USER:-buzz}:${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB:-buzz}
REDIS_URL: redis://:${REDIS_PASSWORD:?set REDIS_PASSWORD}@redis:6379
BUZZ_S3_ENDPOINT: http://minio:9000
# Docker DNS resolves `minio`, not arbitrary `<bucket>.minio` hosts.
BUZZ_S3_ADDRESSING_STYLE: path
BUZZ_S3_ACCESS_KEY: ${BUZZ_S3_ACCESS_KEY:?set BUZZ_S3_ACCESS_KEY}
BUZZ_S3_SECRET_KEY: ${BUZZ_S3_SECRET_KEY:?set BUZZ_S3_SECRET_KEY}
BUZZ_S3_BUCKET: ${BUZZ_S3_BUCKET:-buzz-media}
desktop/src-tauri/src/commands/media_snapshot_png.rs
+1
View File
@@ -204,6 +204,7 @@ mod tests {
s3_secret_key: String::new(),
s3_bucket: String::new(),
s3_region: "us-east-1".to_string(),
s3_addressing_style: buzz_media_pkg::S3AddressingStyle::Path,
max_image_bytes: 50 * 1024 * 1024,
max_gif_bytes: 10 * 1024 * 1024,
max_video_bytes: 524_288_000,