Cloud Storage v2.48.1: Storage.moveBlob throws exception

[wsjlamoureaux]

Description

The new MoveBlob operation fails for non-hierarchical-namespace buckets. We found that the operation fails on buckets with a flat namespace, but succeeds on buckets created with a hierarchical namespace.

The latest cloud storage documentation advertises an atomic move blob: https://cloud.google.com/java/docs/reference/google-cloud-storage/latest/com.google.cloud.storage.Storage#com_google_cloud_storage_Storage_moveBlob_com_google_cloud_storage_Storage_MoveBlobRequest_, but does not mention any restriction in its use.

Environment details

Java 21

Questions

1. Is the documentation incomplete?
2. Will the MoveBlob operation be supported for flat-namespace buckets?
3. Have we missed something?

Code example

package com.example.service

import com.google.cloud.storage.Blob
import com.google.cloud.storage.BlobId
import com.google.cloud.storage.BlobInfo
import com.google.cloud.storage.Storage
import com.google.cloud.storage.StorageException
import com.google.cloud.storage.StorageOptions
import spock.lang.IgnoreIf
import spock.lang.Specification

import java.nio.charset.StandardCharsets

@IgnoreIf(
    value = {!System.getenv('GOOGLE_CLOUD_PROJECT')},
    reason = 'GOOGLE_CLOUD_PROJECT environment variable not set'
)
class MoveBlobTest extends Specification {
    private static final String FLAT_BUCKET_NAME = 'move_blob_test_flat'
    private static final String HIERARCHICAL_BUCKET_NAME = 'move_blob_test_hierarchical'

    // Relying on the GOOGLE_CLOUD_PROJECT environment variable for the project-id value
    private final Storage storage = StorageOptions.newBuilder().build().service

    def "Should fail to atomically move a blob in a flat-namespace bucket"() {
        given: "A source blob"
        Blob sourceBlob = storage.create(
            BlobInfo.newBuilder(FLAT_BUCKET_NAME, 'source-file.txt')
                .setContentType('text/plain')
                .build(),
            'file contents'.getBytes(StandardCharsets.UTF_8)
        )
        and: "A destination blob id"
        BlobId targetBlobId = BlobId.of(FLAT_BUCKET_NAME, 'target-file.txt')

        when: "A move is requested from source to target"
        Blob targetBlob = storage.moveBlob(
            Storage.MoveBlobRequest.newBuilder()
                .setSource(sourceBlob.blobId)
                .setTarget(targetBlobId)
                .build()
        )

        then: "The move operation fails with a StorageException"
        thrown(StorageException)

        and: "The source blob still exists"
        sourceBlob.exists()

        and: "The target blob does not exist"
        !targetBlob?.exists()
    }

    def "Should atomically move a blob in a hierarchical-namespace bucket"() {
        given: "A source blob"
        Blob sourceBlob = storage.create(
            BlobInfo.newBuilder(HIERARCHICAL_BUCKET_NAME, 'source-file.txt')
                .setContentType('text/plain')
                .build(),
            'file contents'.getBytes(StandardCharsets.UTF_8)
        )
        and: "A destination blob id"
        BlobId targetBlobId = BlobId.of(HIERARCHICAL_BUCKET_NAME, 'target-file.txt')

        when: "A move is requested from source to target"
        Blob targetBlob = storage.moveBlob(
            Storage.MoveBlobRequest.newBuilder()
                .setSource(sourceBlob.blobId)
                .setTarget(targetBlobId)
                .build()
        )

        then: "The move operation succeeds and returns a Blob"
        targetBlob

        and: "The source blob no longer exists"
        !sourceBlob.exists()
    }
}

Stack trace

com.google.cloud.storage.StorageException: The bucket does not support hierarchical namespace.
	at com.google.cloud.storage.StorageException.translate(StorageException.java:209)
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:347)
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.moveObject(HttpStorageRpc.java:1195)
	at com.google.cloud.storage.StorageImpl.lambda$moveBlob$73(StorageImpl.java:1711)
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:102)
	at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
	at com.google.cloud.storage.Retrying.run(Retrying.java:65)
	at com.google.cloud.storage.StorageImpl.run(StorageImpl.java:1611)
	at com.google.cloud.storage.StorageImpl.moveBlob(StorageImpl.java:1708)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at com.example.service.MoveBlobTest.$spock_feature_0_0(MoveBlobTest.groovy:38)
        ... more
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 409 Conflict
POST https://storage.googleapis.com/storage/v1/b/move_blob_test_flat/o/source-file.txt/moveTo/o/target-file.txt
{
  "code" : 409,
  "errors" : [ {
    "domain" : "global",
    "message" : "The bucket does not support hierarchical namespace.",
    "reason" : "conflict"
  } ],
  "message" : "The bucket does not support hierarchical namespace."
}
	at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)
	at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
	at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$3.interceptResponse(AbstractGoogleClientRequest.java:479)
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:565)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:506)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:616)
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.moveObject(HttpStorageRpc.java:1193)
	... 102 more

External references such as API reference guides

• https://cloud.google.com/java/docs/reference/google-cloud-storage/latest/com.google.cloud.storage.Storage#com_google_cloud_storage_Storage_moveBlob_com_google_cloud_storage_Storage_MoveBlobRequest_

[ddelgrosso1]

Hi @wsjlamoureaux apologies for the confusion. At present MoveBlob will only work with HNS enabled buckets. Support for non-HNS enabled buckets will be coming in the future. I'll see if we can't get these docs cleaned up to make it clear that at present only HNS enabled buckets will work.