Add supported S3 server access/request logging to IBM Storage Scale S3 (MMS3)

IBM Storage Scale S3 should provide a fully supported production mechanism for S3 server access/request logging.

Storage administrators need request-level visibility comparable to traditional HTTP/S3 gateway access logs without enabling global NooBaa debug logging.

The requested capability should record each completed S3 request and make the records available through an IBM-supported logging mechanism.

At minimum, each access record should contain:

• Timestamp
• HTTP method
• S3 operation
• Source bucket
• Object key / request URI
• Remote client IP
• HTTP response status code
• Request ID
• Authenticated account/principal or access-key identifier where appropriate and safe
• CES/S3 endpoint node that processed the request

The feature should record successful requests as well as failures so administrators can report and monitor response codes such as:

• 2xx
• 3xx
• 4xx
• 5xx

Requested administration

The feature should be configurable and supportable through the normal Storage Scale management interface, preferably mms3.

For example, Storage Scale should provide supported controls to:

• Enable or disable S3 access logging
• Enable it globally or for selected buckets
• Display current logging configuration
• Select the logging destination
• Configure retention/rotation where applicable
• Configure a target S3 bucket/prefix where applicable
• Configure a local file or standard syslog destination where applicable

Configuration should be cluster-aware and persist correctly across CES node restart, failover, upgrade and node replacement.

It should not require manually editing NooBaa source files, systemd units, undocumented environment variables, or IBM-managed configuration files.

Current product behavior / technical background

The NooBaa code shipped with Storage Scale contains implementations of the standard S3 GetBucketLogging and PutBucketLogging APIs as well as bucket access-log record generation.

The implementation can generate structured records containing fields such as:

• operation/method
• source bucket
• object/request URI
• remote IP
• HTTP status
• request ID
• target log bucket
• target log prefix

The shipped code also contains both BEST_EFFORT and PERSISTENT bucket logging implementations.

In the Storage Scale NSFS runtime we tested, the standard S3 Bucket Logging APIs are callable and accept a bucket logging configuration. However, the default BEST_EFFORT path is not usable because the NSFS object SDK does not provide the RPC client expected by that logging path.

There is also a persistent WAL/export implementation in the shipped code, but there is no documented Storage Scale/mms3 method to configure and operate it as a supported S3 access-logging feature.

A separate IBM Support case is being opened for the current Bucket Logging API behavior.

Business justification

S3 gateways are increasingly used for large-scale production workloads where hundreds or thousands of concurrent object requests may be active.

Storage administrators need to answer questions such as:

• Which clients are receiving 4xx or 5xx responses?
• Which bucket or object is generating failures?
• Did a specific request reach the S3 gateway?
• Which CES S3 node handled a request?
• What HTTP return codes are being generated over a time period?
• Are errors isolated to a client, bucket, operation or gateway node?
• What percentage of requests are successful versus unsuccessful?

Aggregate metrics and error-only application logs cannot answer all of these questions.

Enabling DEBUGLEVEL=all is also not an appropriate substitute for a production access log because it enables broad application debugging and can generate very large amounts of unrelated log data.