Vfrdtyky

I made a spark code that makes a copy of a folder and put it to an Amazon S3 standalone bucket. The process works just fine but now I'm trying to use the same process with an Amazon S3 bucket that runs over Scality. This is my configuration.

spark-submit --name "Backup S3 Test" --master yarn-cluster  --executor-memory 2048m --num-executors 6 --executor-cores 2 --driver-memory 1024m --keytab /home/bigdata/userbcks3.keytab 

--principal XXXXXXX@XXXXXXXX 

--deploy-mode cluster 

--conf spark.file.replicate.exclusion.regexps="" 

--conf spark.hadoop.fs.s3a.access.key=XXXXXXXXXX 

--conf spark.hadoop.fs.s3a.secret.key=XXXXXXXXXX 

--class com.keedio.hadoop.FileReplicator hdfs-file-processors-1.1.6-SNAPSHOT.jar /pre/mydata/ s3a://mybucket/

And now the exception

om.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)



        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)



        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)



        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)



        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4221)



        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4168)



        at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1306)



        at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1263)



        at org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:323)



        ... 20 more



Caused by: com.amazonaws.SdkClientException: The requested metadata is not found at http://169.254.169.254/latest/meta-data/iam/security-credentials/



        at com.amazonaws.internal.EC2CredentialsUtils.readResource(EC2CredentialsUtils.java:115)



        at com.amazonaws.internal.EC2CredentialsUtils.readResource(EC2CredentialsUtils.java:77)



        at com.amazonaws.auth.InstanceProfileCredentialsProvider$InstanceMetadataCredentialsEndpointProvider.getCredentialsEndpoint(InstanceProfileCredentialsProvider.java:156)



        at com.amazonaws.auth.EC2CredentialsFetcher.fetchCredentials(EC2CredentialsFetcher.java:121)



        at com.amazonaws.auth.EC2CredentialsFetcher.getCredentials(EC2CredentialsFetcher.java:82)



        at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:141)



        at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:129)

To make the copy i'm just using apache FileUtils that allow me to move files between DistributedFileSystem and S3AFileSystem.
Is there any way i can make it work with the same process? Maybe is there any configuration parameter I'm missing?

whatever program you are running, it's not picked up the fs.s3a.access.key/secret.key values, is running through the other auth options (env vars, EC2 metadata server) and failing. You haven't got as far as communications with the far end yet.

if you code worked before, and it was running in EC2, then it may always have been that metadata server which was logging you in...