bpython怎么保存_python - 如何使用b将文件上传到S3存储桶中的目录

python - 如何使用b将文件上传到S3存储桶中的目录

我想使用python在s3存储桶中复制一个文件。

例如：我有桶名=测试。在桶中，我有2个文件夹名称“dump”＆amp;“输入”。现在我想使用python将文件从本地目录复制到S3“dump”文件夹...任何人都可以帮助我吗？

10个解决方案

81 votes

试试这个...

import boto

import boto.s3

import sys

from boto.s3.key import Key

AWS_ACCESS_KEY_ID = ''

AWS_SECRET_ACCESS_KEY = ''

bucket_name = AWS_ACCESS_KEY_ID.lower() + '-dump'

conn = boto.connect_s3(AWS_ACCESS_KEY_ID,

AWS_SECRET_ACCESS_KEY)

bucket = conn.create_bucket(bucket_name,

location=boto.s3.connection.Location.DEFAULT)

testfile = "replace this with an actual filename"

print 'Uploading %s to Amazon S3 bucket %s' % \

(testfile, bucket_name)

def percent_cb(complete, total):

sys.stdout.write('.')

sys.stdout.flush()

k = Key(bucket)

k.key = 'my test file'

k.set_contents_from_filename(testfile,

cb=percent_cb, num_cb=10)

[UPDATE]我不是pythonist，所以感谢关于import语句的提醒。另外，我不建议在自己的源代码中放置凭据。如果您在AWS内部运行，请使用带有实例配置文件的IAM凭据([http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html]，并保持不变您的开发/测试环境中的行为，使用来自AdRoll的全息图([https://github.com/AdRoll/hologram]]

Felipe Garcia answered 2019-06-14T21:32:32Z

36 votes

不需要那么复杂：

s3_connection = boto.connect_s3()

bucket = s3_connection.get_bucket('your bucket name')

key = boto.s3.key.Key(bucket, 'some_file.zip')

with open('some_file.zip') as f:

key.send_file(f)

vcarel answered 2019-06-14T21:32:56Z

31 votes

我用过它，实现起来非常简单

import tinys3

conn = tinys3.Connection('S3_ACCESS_KEY','S3_SECRET_KEY',tls=True)

f = open('some_file.zip','rb')

conn.upload('some_file.zip',f,'my_bucket')

[https://www.smore.com/labs/tinys3/]

Oren Efron answered 2019-06-14T21:33:26Z

15 votes

import boto3

s3 = boto3.resource('s3')

BUCKET = "test"

s3.Bucket(BUCKET).upload_file("your/local/file", "dump/file")

Boris answered 2019-06-14T21:33:43Z

10 votes

这也有效：

import os

import boto

import boto.s3.connection

from boto.s3.key import Key

try:

conn = boto.s3.connect_to_region('us-east-1',

aws_access_key_id = 'AWS-Access-Key',

aws_secret_access_key = 'AWS-Secrete-Key',

# host = 's3-website-us-east-1.amazonaws.com',

# is_secure=True, # uncomment if you are not using ssl

calling_format = boto.s3.connection.OrdinaryCallingFormat(),

)

bucket = conn.get_bucket('YourBucketName')

key_name = 'FileToUpload'

path = 'images/holiday' #Directory Under which file should get upload

full_key_name = os.path.join(path, key_name)

k = bucket.new_key(full_key_name)

k.set_contents_from_filename(key_name)

except Exception,e:

print str(e)

print "error"

Piyush S. Wanare answered 2019-06-14T21:34:01Z

9 votes

from boto3.s3.transfer import S3Transfer

import boto3

#have all the variables populated which are required below

client = boto3.client('s3', aws_access_key_id=access_key,aws_secret_access_key=secret_key)

transfer = S3Transfer(client)

transfer.upload_file(filepath, bucket_name, folder_name+"/"+filename)

Manish Mehra answered 2019-06-14T21:34:18Z

4 votes

import boto

from boto.s3.key import Key

AWS_ACCESS_KEY_ID = ''

AWS_SECRET_ACCESS_KEY = ''

END_POINT = '' # eg. us-east-1

S3_HOST = '' # eg. s3.us-east-1.amazonaws.com

BUCKET_NAME = 'test'

FILENAME = 'upload.txt'

UPLOADED_FILENAME = 'dumps/upload.txt'

# include folders in file path. If it doesn't exist, it will be created

s3 = boto.s3.connect_to_region(END_POINT,

aws_access_key_id=AWS_ACCESS_KEY_ID,

aws_secret_access_key=AWS_SECRET_ACCESS_KEY,

host=S3_HOST)

bucket = s3.get_bucket(BUCKET_NAME)

k = Key(bucket)

k.key = UPLOADED_FILENAME

k.set_contents_from_filename(FILENAME)

Shakti answered 2019-06-14T21:34:35Z

0 votes

xmlstr = etree.tostring(listings, encoding='utf8', method='xml')

conn = boto.connect_s3(

aws_access_key_id = access_key,

aws_secret_access_key = secret_key,

# host = '.s3.amazonaws.com',

host = 'bycket.s3.amazonaws.com',

#is_secure=False, # uncomment if you are not using ssl

calling_format = boto.s3.connection.OrdinaryCallingFormat(),

)

conn.auth_region_name = 'us-west-1'

bucket = conn.get_bucket('resources', validate=False)

key= bucket.get_key('filename.txt')

key.set_contents_from_string("SAMPLE TEXT")

key.set_canned_acl('public-read')

Martin answered 2019-06-14T21:34:51Z

0 votes

这是一个三班轮。只需按照boto3文档中的说明操作即可。

import boto3

s3 = boto3.resource(service_name = 's3')

s3.meta.client.upload_file(Filename = 'C:/foo/bar/baz.filetype', Bucket = 'yourbucketname', Key = 'baz.filetype')

一些重要的论点是：

参数：

文件名(~\.aws) - 要上载的文件的路径。

Bucket(~\.aws) - 要上载到的存储桶的名称。

密钥(~\.aws) - 要在s3存储桶中分配给文件的名称。这可能与文件名或您选择的其他名称相同，但文件类型应保持不变。

注意：我假设您已将凭据保存在~\.aws文件夹中，如boto3文档中的最佳配置实践中所述。

Nde Samuel Mbah answered 2019-06-14T21:35:56Z

0 votes

对于上传文件夹示例如下代码和S3文件夹图片

import boto

import boto.s3

import boto.s3.connection

import os.path

import sys

# Fill in info on data to upload

# destination bucket name

bucket_name = 'willie20181121'

# source directory

sourceDir = '/home/willie/Desktop/x/' #Linux Path

# destination directory name (on s3)

destDir = '/test1/' 'S3 Path

#max size in bytes before uploading in parts. between 1 and 5 GB recommended

MAX_SIZE = 20 * 1000 * 1000

#size of parts when uploading in parts

PART_SIZE = 6 * 1000 * 1000

access_key = 'MPBVAQ*******IT****'

secret_key = '11t63yDV***********HgUcgMOSN*****'

conn = boto.connect_s3(

aws_access_key_id = access_key,

aws_secret_access_key = secret_key,

host = '******.org.tw',

is_secure=False, # uncomment if you are not using ssl

calling_format = boto.s3.connection.OrdinaryCallingFormat(),

)

bucket = conn.create_bucket(bucket_name,

location=boto.s3.connection.Location.DEFAULT)

uploadFileNames = []

for (sourceDir, dirname, filename) in os.walk(sourceDir):

uploadFileNames.extend(filename)

break

def percent_cb(complete, total):

sys.stdout.write('.')

sys.stdout.flush()

for filename in uploadFileNames:

sourcepath = os.path.join(sourceDir + filename)

destpath = os.path.join(destDir, filename)

print ('Uploading %s to Amazon S3 bucket %s' % \

(sourcepath, bucket_name))

filesize = os.path.getsize(sourcepath)

if filesize > MAX_SIZE:

print ("multipart upload")

mp = bucket.initiate_multipart_upload(destpath)

fp = open(sourcepath,'rb')

fp_num = 0

while (fp.tell() < filesize):

fp_num += 1

print ("uploading part %i" %fp_num)

mp.upload_part_from_file(fp, fp_num, cb=percent_cb, num_cb=10, size=PART_SIZE)

mp.complete_upload()

else:

print ("singlepart upload")

k = boto.s3.key.Key(bucket)

k.key = destpath

k.set_contents_from_filename(sourcepath,

cb=percent_cb, num_cb=10)

PS：有关更多参考URL

Willie Cheng answered 2019-06-14T21:36:26Z