MySql update query - Update column by string replacement in all records
Problem Statement In a mysql table, I wanted to replace the hostname of the…
February 04, 2021
In this tutorial we will see, How to list and download storage container blobs without using Azure python libraries.
Note: There is no azure library used, just rest api calls.
This tutorial is based upon Python-3.7
We would require requests
.
import requests
import re
import os
def _get_file_list_helper(container, next_marker=None):
"""
Get the files list by using next_marker
"""
account_name = container['account_name']
container_name = container['container_name']
curl_url = f'https://{account_name}.blob.core.windows.net/{container_name}?restype=container&comp=list&'
if next_marker:
curl_url += f'marker={next_marker}&'
curl_url += container['sas_token']
print('Executing rest call to azure')
r = requests.get(curl_url)
text = r.text
# this marker indicates there are more files
next_marker = re.findall('<NextMarker>([^<]*)</NextMarker>',text)
file_names = re.findall('<Name>([^<]*)</Name>',text)
return {'files': file_names, 'next_marker': next_marker}
def get_file_list(container):
"""
Get the files list
"""
files = []
next_marker = None
while True:
files_data = _get_file_list_helper(container, next_marker)
files.extend(files_data['files'])
if not files_data['next_marker']:
break
next_marker = files_data['next_marker'][0]
return files
def dowload_files(container, local_dest_path):
files = get_file_list(container)
account_name = container['account_name']
container_name = container['container_name']
url_path = f'https://{account_name}.blob.core.windows.net/{container_name}/'
url_end_path = '?' + container['sas_token']
for file_name in files:
print(f'Downloading: {file_name}')
url = f'{url_path}{file_name}{url_end_path}'
path = f'{local_dest_path}/{file_name}'
if not os.path.exists(os.path.dirname(path)):
os.makedirs(os.path.dirname(path))
# make the request
r = requests.get(url)
# write the file
with open(path, "wb") as download_file:
download_file.write(r.content)
## main starts here
local_dest_path = './container_blob'
container = {
'account_name': 'account_name',
'container_name': 'container_name',
'sas_token': 'xxxxxxxxxx'
}
dowload_files(container, local_dest_path)
The code is very simple to understand. We are using Azure REST APIs to list and download storage blobs.
In cases, where there are more files in your storage container. The response does not have all the files in one response call. It instead returns a fixed number of items and a next_marker
. Which indicates, there are more files. This marker has to be sent in next requests.
For usage with Azure official Python libraries, see: List and Download Azure blobs by Azure Python Libraries
<?xml version="1.0" encoding="utf-8"?><EnumerationResults ServiceEndpoint="https://hubbledmeprodlocb.blob.core.windows.net/" ContainerName="container_name">
<Blobs>
<Blob>
<Name>abc/test.log</Name>
<Properties>
<Last-Modified>Mon, 02 Dec 2019 09:42:50 GMT</Last-Modified>
<Etag>0x8D7770BFF1CC8A1</Etag>
<Content-Length>423</Content-Length>
<Content-Type>application/octet-stream</Content-Type>
<Content-Encoding /><Content-Language />
<Content-MD5>3ycLC3CutKkybJtlgvEdsQ==</Content-MD5>
<Cache-Control />
<Content-Disposition />
<BlobType>BlockBlob</BlobType>
<LeaseStatus>unlocked</LeaseStatus>
<LeaseState>available</LeaseState>
<ServerEncrypted>true</ServerEncrypted>
</Properties>
</Blob>
...
</Blobs>
<NextMarker>marker_id</NextMarker>
</EnumerationResults>
Hope it helps.
Problem Statement In a mysql table, I wanted to replace the hostname of the…
While doing code review of one of junior, I encountered following code: What…
Introduction In this guide, We will learn about branching, handling conflict…
Bootstrap has a simple solution to have sticky blocks in your html. I’ve given a…
Introduction This post has the complete code to send email through smtp server…
This post some useful tips of using strings, and some issues while dealing with…
Introduction This post has the complete code to send email through smtp server…
Introduction In a normal email sending code from python, I’m getting following…
Introduction In one of my app, I was using to talk to . I have used some event…
Introduction So you have a Django project, and want to run it using docker image…
Introduction It is very important to introduce few process so that your code and…
Introduction In this post, we will see a sample Jenkin Pipeline Groovy script…