Scenario:
Download excel file from Azure file system and process it in server through Node Js to generate some meaningful reports.
Pre-requisite:
Here, I am going to assume that as a developer, we are already having below information or tasks ready to write required code to fix the above scenario:
- A running node js application where we can add required code to read files from Azure file system and download it to our server where node js is running.
- Install package like: @azure/storage-file-share
- Required credentials to connect with your Azure storage like AccountName and AccountKey
Note: Here, I am presenting only the way of getting excel files from Azure file system and saving it in your server. To see, how your NodeJs code can read these files from your node js server and process every sheets of every excel file, you can go through my another blog:
Implementation:
1) This implementation can be done in following steps:Create azure connection string: Keep your Azure AccountName and AccountKey somewhere safe like in your environment variable or DB. Create connection string by using these two data in following way:
"azureStorageConnectionString": "DefaultEndpointsProtocol=https;AccountName=YourAccountName;AccountKey=YourAccountKey;EndpointSuffix=core.windows.net",
2) Create a temp directory in your server where your node js code is running so that your node js code can further read and process the downloaded files from this temp directory.
const tempDir = path.join(__dirname, '../../temp_dir_name');
//creating temp folder path
fs.mkdirSync(tempDir, { recursive: true })
3) Write code for downloading file from Azure file system.
const azurefileClient = serviceClient
.getShareClient(shareName)
.rootDirectoryClient.getFileClient(filePathInAzureFileSystem);
// Get file content from azure file system
const azureFileResponse = await azurefileClient.download();
azureFileResponse.readableStreamBody.on('end', () => {
console.log("File downloaded successfully but yet to be written on nodejs server");
});
Here, we are creating a read stream i.e. azureFileResponse for reading the file (represented by filePathInAzureFileSystem having the path as well as file name like ext-files/abc/xyz/readMe.xlsx) in binary format from azure file sytem. We are attaching one event handler on this read stream to notify once reading is completed.
4) Now, we are going to create one write stream that will take input from the above read stream and write the data in the temp folder in the same file name in nodejs server.
//creating write stream
let writer = fs.createWriteStream(tempDir + "/" + "fileName.xlsx");
writer.on('error', (err) => {
console.log(err, "Error in writing the downloaded file in nodejs server");
});
Here, what ever data is being read from azure read stream, that data will be written to the file tempDir/fileName.xlsx file in node js server. Here, we have attached an error event handler with temp file write stream to handle any error if it happens during writing data to temp file.
5) Connect the azure file read stream with temp file write stream through node js pipe functionality so that temp file write stream will receive chucks of data from azure read stream and will keep writing to temp file in nodejs env.
//connecting readStream with write Stream to write downloaded file into temp folder in node js server
return await pipeline(azureFileResponse.readableStreamBody, writer);
Following is the complete code:
const { ShareServiceClient } = require("@azure/storage-file-share");
const fs = require('fs');
const path = require('path');
const stream = require('stream');
const pipeline = util.promisify(stream.pipeline);
const shareName = config.shareName;
const tempDir = path.join(__dirname, '../../temp_dir_name');
const azureStorageConnectionStr = "DefaultEndpointsProtocol=https;AccountName=YourAccountName;AccountKey=YourAccountKey;EndpointSuffix=core.windows.net";
const serviceClient = ShareServiceClient.fromConnectionString(azureStorageConnectionStr);
const downloadFileFromAzureFileSystem = async (filePathInAzureFileSystem) => {
try {
//creating temp folder path
fs.mkdirSync(tempDir, { recursive: true })
const azurefileClient = serviceClient
.getShareClient(shareName)
.rootDirectoryClient.getFileClient(filePathInAzureFileSystem);
// Get file content from azure file system
const azureFileResponse = await azurefileClient.download();
azureFileResponse.readableStreamBody.on('end', () => {
console.log("File downloaded successfully but yet to be written on nodejs server");
});
//creating write stream
let tempFileWriter = fs.createWriteStream(tempDir + "/" + "fileName.xlsx");
tempFileWriter.on('error', (err) => {
console.log(err, "Error in writing the downloaded file in nodejs server");
});
//connecting readStream with write Stream to write downloaded file into temp folder in node js server
return await pipeline(azureFileResponse.readableStreamBody, tempFileWriter);
} catch (error) {
console.log(error);
throw error;
}
}
module.exports = {
downloadFileFromAzureFileSystem
}
