IBM Storage Scale S3 should provide a fully supported production mechanism for S3 server access/request logging.
Storage administrators need request-level visibility comparable to traditional HTTP/S3 gateway access logs without enabling global NooBaa debug logging.
The requested capability should record each completed S3 request and make the records available through an IBM-supported logging mechanism.
At minimum, each access record should contain:
- Timestamp
- HTTP method
- S3 operation
- Source bucket
- Object key / request URI
- Remote client IP
- HTTP response status code
- Request ID
- Authenticated account/principal or access-key identifier where appropriate and safe
- CES/S3 endpoint node that processed the request
The feature should record successful requests as well as failures so administrators can report and monitor response codes such as:
Requested administration
The feature should be configurable and supportable through the normal Storage Scale management interface, preferably mms3.
For example, Storage Scale should provide supported controls to:
- Enable or disable S3 access logging
- Enable it globally or for selected buckets
- Display current logging configuration
- Select the logging destination
- Configure retention/rotation where applicable
- Configure a target S3 bucket/prefix where applicable
- Configure a local file or standard syslog destination where applicable
Configuration should be cluster-aware and persist correctly across CES node restart, failover, upgrade and node replacement.
It should not require manually editing NooBaa source files, systemd units, undocumented environment variables, or IBM-managed configuration files.
Current product behavior / technical background
The NooBaa code shipped with Storage Scale contains implementations of the standard S3 GetBucketLogging and PutBucketLogging APIs as well as bucket access-log record generation.
The implementation can generate structured records containing fields such as:
- operation/method
- source bucket
- object/request URI
- remote IP
- HTTP status
- request ID
- target log bucket
- target log prefix
The shipped code also contains both BEST_EFFORT and PERSISTENT bucket logging implementations.
In the Storage Scale NSFS runtime we tested, the standard S3 Bucket Logging APIs are callable and accept a bucket logging configuration. However, the default BEST_EFFORT path is not usable because the NSFS object SDK does not provide the RPC client expected by that logging path.
There is also a persistent WAL/export implementation in the shipped code, but there is no documented Storage Scale/mms3 method to configure and operate it as a supported S3 access-logging feature.
A separate IBM Support case is being opened for the current Bucket Logging API behavior.
Business justification
S3 gateways are increasingly used for large-scale production workloads where hundreds or thousands of concurrent object requests may be active.
Storage administrators need to answer questions such as:
- Which clients are receiving 4xx or 5xx responses?
- Which bucket or object is generating failures?
- Did a specific request reach the S3 gateway?
- Which CES S3 node handled a request?
- What HTTP return codes are being generated over a time period?
- Are errors isolated to a client, bucket, operation or gateway node?
- What percentage of requests are successful versus unsuccessful?
Aggregate metrics and error-only application logs cannot answer all of these questions.
Enabling DEBUGLEVEL=all is also not an appropriate substitute for a production access log because it enables broad application debugging and can generate very large amounts of unrelated log data.
