Azure Databricks : Save file in Azure Datalake directory folder

Question

I would like to save file in my azure datalake gen2 directory by it is not working.

When I save the file without directory it works but with Directory "Data" it is not working.

Here is the code :

# Create a file in local data directory to upload and download local_path = "./Data" local_file_name = "quickstart" + str(uuid.uuid4()) + ".txt" upload_file_path = os.path.join(local_path, local_file_name) # Write text to the file file = open(upload_file_path, 'w') file.write("Hello, World!") file.close() # Create a blob client using the local file name as the name for the blob blob_client = blob_service_client.get_blob_client(container=container_name, blob=local_file_name) print("\nUploading to Azure Storage as blob:\n\t" + local_file_name) # Upload the created file with open(upload_file_path, "rb") as data: blob_client.upload_blob(data)

Here is the error:

FileNotFoundError: [Errno 2] No such file or directory: './Data/quickstart564851de-67a3-4b28-9af1-049298f26408.txt' --------------------------------------------------------------------------- FileNotFoundError Traceback (most recent call last) <command-2012583145450168> in <module> 5 6 # Write text to the file ----> 7 file = open(upload_file_path, 'w') 8 file.write("Hello, World!") 9 file.close() FileNotFoundError: [Errno 2] No such file or directory: './Data/quickstart564851de-67a3-4b28-9af1-049298f26408.txt'

The folder "Data" exist in my data lake :

from azure.storage.filedatalake import FileSystemClient file_system = FileSystemClient.from_connection_string(connect_str, file_system_name="flatdata") paths = file_system.get_paths() for path in paths: print(path.name + '\t')

output :

Data doto new.csv quickstart.csv quickstartc0afe722-3851-4f5a-a6fa-83fd97389c43.txt saved.csv

Here is the documentation from microsoft : click here to see the Documentation

Thanks for your help

You can Upload a file to a directory. Is this helpful for you? — Joseph Xu
– Joseph Xu, Commented Jan 6, 2021 at 8:37
Hi @hugo, since you run your code in azure databaricks, why do you not use pyspark? You can directly save file to data lake gen2. — Jim Xu
– Jim Xu, Commented Jan 7, 2021 at 2:06

Joseph Xu · Accepted Answer · 2021-01-07 01:12:54Z

Reference to the documentation. I've created a test to upload local file to a folder of my Azure data lake.
This is my local project file structure:

The file was uploaded to the folder of my Azure data lake.

This is my python code

 import os, uuid from azure.storage.filedatalake import DataLakeServiceClient # Create a file in local data directory to upload and download local_path = "./data" local_file_name = "quickstart" + str(uuid.uuid4()) + ".txt" upload_file_path = os.path.join(local_path, local_file_name) # Write text to the file file = open(upload_file_path, 'w') file.write("Hello, World!") file.close() #upload the file to the folder file_system_client = service_client.get_file_system_client(file_system="test10") directory_client = file_system_client.get_directory_client("data") file_client = directory_client.create_file(local_file_name) local_file = open(upload_file_path, 'rb') file_contents = local_file.read() file_client.append_data(data=file_contents, offset=0, length=len(file_contents)) file_client.flush_data(len(file_contents))

I have this error : FileNotFoundError: [Errno 2] No such file or directory: './data/quickstart2d78e9d2-8cbc-4bbc-a43d-a3e33ce037b3.txt' --------------------------------------------------------------------------- FileNotFoundError Traceback (most recent call last) <command-2416439453149044> in <module> 8 9 # Write text to the file ---> 10 file = open(upload_file_path, 'w') 11 file.write("Hello, World!") 12 file.close() FileNotFoundError: [Errno 2] No such file or directory: './data/quickstart2d78e9d2-8cbc-4bbc-a43d-a3e33ce037b3.txt'
Hi @hugo, please see my first picture is shows the local project file structure. There is a data folder under the relative path of my upload_ADL.py. Have you created the data folder in your local project ?
Hi @hugo Kindly let me know if you need more information. : )

Collectives™ on Stack Overflow

Azure Databricks : Save file in Azure Datalake directory folder

1 Answer 1

3 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Related