Thursday, September 9, 2021

How to create a Post request with Multipart header to upload a file through Postman

 Scenario:

Create a Post Request to upload two files along with some input data through Postman.

Note: Handling of this Post Request for file uploading will be discussed in next Blog.

Steps:

  • Open Postman and create a Post request            
  • Click on the Body tab and then select form-data radio button

  • Enter the field name like "myfile1" in textbox in column with Header KEY and then select "File" from the drop down

  • Once you select File from the dropdown, you will get a button "Select Files" under the column VALUE. You can click on this button and can select the file that you want to upload.


  • Now, suppose you want to send name and emailId and input data with this Post request. So, you should add the field names like "name" and "emailId" in KEY column and corresponding values in VALUE column. Here, by default, TEXT remains selected.






Sunday, August 29, 2021

Download file from Azure file and save it in temp folder in server by using Node js

 Scenario

Download excel file from Azure file system and process it in server through Node Js to generate some meaningful reports.

Pre-requisite:

Here, I am going to assume that as a developer, we are already having below information or tasks ready to write required code to fix the above scenario:

  1. A running node js application where we can add required code to read files from Azure file system and download it to our server where node js is running.
  2. Install package like: @azure/storage-file-share
  3. Required credentials to connect with your Azure storage like AccountName and AccountKey
Note: Here, I am presenting only the way of getting excel files from Azure file system and saving it in your server. To see, how your NodeJs code can read these files from your node js server and process every sheets of every excel file, you can go through my another blog:


Implementation:

1) This implementation can be done in following steps:Create azure connection string: Keep your Azure AccountName and AccountKey somewhere safe like in your environment variable or DB. Create connection string by using these two data in following way: 

"azureStorageConnectionString""DefaultEndpointsProtocol=https;AccountName=YourAccountName;AccountKey=YourAccountKey;EndpointSuffix=core.windows.net",

2) Create a temp directory in your server where your node js code is running so that your node js code can further read and process the downloaded files from this temp directory.

const tempDir = path.join(__dirname'../../temp_dir_name');
//creating temp folder path 
fs.mkdirSync(tempDir, { recursive: true })

3) Write code for downloading file from Azure file system.

const azurefileClient = serviceClient
    .getShareClient(shareName)
    .rootDirectoryClient.getFileClient(filePathInAzureFileSystem);
    // Get file content from azure file system
    const azureFileResponse = await azurefileClient.download();
    azureFileResponse.readableStreamBody.on('end', () => {
      console.log("File downloaded successfully but yet to be written on nodejs server");
    });

Here, we are creating a read stream i.e. azureFileResponse for reading the file (represented by filePathInAzureFileSystem having the path as well as file name like ext-files/abc/xyz/readMe.xlsx) in binary format from azure file sytem. We are attaching one event handler on this read stream to notify once reading is completed.

4) Now, we are going to create one write stream that will take input from the above read stream and write the data in the temp folder in the same file name in nodejs server.

//creating write stream
    let writer = fs.createWriteStream(tempDir + "/" + "fileName.xlsx");
    writer.on('error', (err=> {
      console.log(err"Error in writing the downloaded file in nodejs server");
    });

Here, what ever data is being read from azure read stream, that data will be written to the file tempDir/fileName.xlsx file in node js server. Here, we have attached an error event handler with temp file write stream to handle any error if it happens during writing data to temp file.

5) Connect the azure file read stream with temp file write stream through node js pipe functionality so that temp file write stream will receive chucks of data from azure read stream and will keep writing to temp file in nodejs env.

//connecting readStream with write Stream to write downloaded file into temp folder in node js server
    return await pipeline(azureFileResponse.readableStreamBodywriter);

Following is the complete code:

const { ShareServiceClient } = require("@azure/storage-file-share");
const fs = require('fs');
const path = require('path');
const stream = require('stream');
const pipeline = util.promisify(stream.pipeline);
const shareName = config.shareName
const tempDir = path.join(__dirname'../../temp_dir_name');
const azureStorageConnectionStr = "DefaultEndpointsProtocol=https;AccountName=YourAccountName;AccountKey=YourAccountKey;EndpointSuffix=core.windows.net";
const serviceClient =  ShareServiceClient.fromConnectionString(azureStorageConnectionStr);

const downloadFileFromAzureFileSystem = async (filePathInAzureFileSystem=> {
  try {   
    //creating temp folder path 
    fs.mkdirSync(tempDir, { recursive: true })

    const azurefileClient = serviceClient
    .getShareClient(shareName)
    .rootDirectoryClient.getFileClient(filePathInAzureFileSystem);
    // Get file content from azure file system
    const azureFileResponse = await azurefileClient.download();
    azureFileResponse.readableStreamBody.on('end', () => {
      console.log("File downloaded successfully but yet to be written on nodejs server");
    });

    //creating write stream
    let tempFileWriter = fs.createWriteStream(tempDir + "/" + "fileName.xlsx");
    tempFileWriter.on('error', (err=> {
      console.log(err"Error in writing the downloaded file in nodejs server");
    });

    //connecting readStream with write Stream to write downloaded file into temp folder in node js server
    return await pipeline(azureFileResponse.readableStreamBodytempFileWriter);
  } catch (error) {
    console.log(error);
    throw error;
  }
}

module.exports = {
  downloadFileFromAzureFileSystem
}


Wednesday, July 21, 2021

Read all sheets of an excel file in NodeJs using XLSX package

Scenario
We have an excel file in our application in a temp folder and we need to process data of each sheet of the excel file. I am giving here the screenshots of folder path of our excel file "candidate.xlsx" with its content.


Here, We need to process data from each sheets of excel file candidate.xlsx present under folder "temp_file".  Below are the screenshots of its two sheets named as "candidates" and "address".



Note: In this sheet, the main headers and data like Name, Email, Phone, Date of birth are starting from 3rd row considering first row as 0. So, header start index is 2. This is going to be used later in code where you can understand it.


Note: In this sheet, the main headers and data like Country, State, City are starting from 4rth row considering first row as 0. So, header start index is 3. This is going to be used later in code where you can understand it.

Solution
There are so many npm packages available to process excel file in Javascript but I found XLSX as more reliable and robust. I will write another blog about how to select the best npm package among several available packages.

The task of processing data of both sheets of above excel file is being performed in following steps:

Step 1:
Installed XLSX package by using command like npm install xlsx

{
  "name""node_excel_read",
  "version""1.0.0",
  "description""",
  "main""index.js",
  "scripts": {
    "test""echo \"Error: no test specified\" && exit 1"
  },
  "author""Jitendra Kumar Singh",
  "license""ISC",
  "dependencies": {
    "xlsx""^0.17.0"
  }
}

Step 2:
Get the list of names of all the excel files present in folder "temp_file". Here, I am not recursively traversing in "temp_file" and I am finding the list of file names present directly inside "temp_file".

/**
 * Creating the list of names of files present directly in temp_file dir
 * @returns 
 */
const getFiles = () => {
  try {
    let files = Array();
    fs.readdirSync(sourceDirPath).forEach(file => {
      if (!fs.lstatSync(path.resolve(sourceDirPathfile)).isDirectory()) {
        files.push(file);
      }
    });
    return files;
  } catch (error) {
    throw error;
  }  
}

Step 3:
Here, I am going to process only one excel file present inside "temp_file" folder. I will write another blog for recursively traversing "temp_file" folder to get list of all file names and processing all files. Now, I am going to read the excel file by using "xlsx" package.

/**
 * Getting the list of name of files present directly in temp_file folder
 * Iterating this list of name of files and then processing each sheet in each file
 * Each sheet is being processed by corresponding function
 * @returns 
 */
const readTestFile = async () => {
  try {
    const files = getFiles();
    if(files && files.length === 0) {
      console.log("No file present");
      return true;
    }
    const filePath = sourceDirPath + "/" + files[0];
    //Reading the excel file candidate.xlsx
    const file = xlsx.readFile(filePath);
    if(file && file.SheetNames && file.SheetNames.length > 0) {
      for(let sheetName of file.SheetNames) {
        if(sheetName === 'candidates') {
          processCandidateSheet(file.Sheets[sheetName]);
        } else if(sheetName === 'address') {
          processAddressSheet(file.Sheets[sheetName]);
        }
      }
    }
  } catch (error) {
    throw error;
  }  
}

Here, I am reading the "candidate.xlsx" file by 
const file = xlsx.readFile(filePath);

Here, the variable "file" is a list of sheets present in this excel file. I am iterating this "file" variable and processing both the sheets "candidates" and "address" by functions "processCandidateSheet()" and "processAddressSheet()"

Step 4:
Here, I am processing "candidates" sheet.

/**
 * Processing the data of Candidate sheet
 * @param {*} candidateSheet 
 */
const processCandidateSheet = (candidateSheet=> {
  try {
    const headerStartIndexInCandidateSheet = 2;//based on position of header in the sheet
    let candidateJsonData = convertSheetToJson(candidateSheetheaderStartIndexInCandidateSheet);
    for(let data of candidateJsonData) {
      console.log(data);
    }
  } catch (error) {
    throw error;
  }
}

If you go through this code block, here I am calling one function convertSheetToJson(). This function actually converts the data of the sheet in JSON format so that it becomes very easy to play with the resultant JSON data. 

/**
 * Converting sheet data in JSON format
 * @param {} sheetName 
 * @param {*} headerIndex 
 * @returns 
 */
const convertSheetToJson = (sheetNameheaderIndex=> {
  try {
    let config = {
      header: "A"
    }
    let sheetData = xlsx.utils.sheet_to_json(sheetNameconfig);
    if (sheetData && sheetData.length > 0) {
      const header = sheetData[headerIndex];
      let loopIndex = headerIndex + 1;
      let jsonRows = Array();
      for (loopIndexloopIndex < sheetData.lengthloopIndex++) {
        let tempObj = {};
        for (const property in sheetData[loopIndex]) {
          tempObj[header[property]] = sheetData[loopIndex][property];
        }
        jsonRows.push(tempObj);
      }
      return jsonRows;
    }
  } catch (error) {
    throw error;
  }
}

Step 5:
Here, I am processing the "address" sheet.

/**
 * Processing the data of Address sheet
 * @param {*} addressSheet 
 */
 const processAddressSheet = (addressSheet=> {
  try {
    const headerStartIndexInAddressSheet = 3;//based on position of header in the sheet
    let addressJsonData = convertSheetToJson(addressSheetheaderStartIndexInAddressSheet);
    for(let data of addressJsonData) {
      console.log(data);
    }
  } catch (error) {
    throw error;
  }
}

Step 6:
Now, I am writing here the function that I will trigger from NodeJs terminal to read file.

const initiate = async () => {
  await readTestFile();
}

setTimeout(() => {
  initiate();
}, 2000);

Step 7:
All the above code I have written in a file named as "readTestFile.js". Here, I have imported following dependencies.

const fs = require('fs');
const path = require('path');
const xlsx = require('xlsx');
const sourceDirPath = path.join(__dirname'../../temp_file');

To run this file I am using command like "node readTestFile"

Output:
Once I run the command like "node readTestFile", it triggers the function "readTestFile()" and starts reading both the sheets of the excel file and printing the data of both the sheets in JSON format.

D:\jitendra\project\poc\node_excel_read\src\scriptsnode readTestFile

Data from candidates sheet

{
  Name'AAA AA',       
  Email'aaa@aa.com',  
  Phone1111111111,    
  'Date of birth'29504
}
{
  Name'BBB BB',
  Email'bbb@bb.com',
  Phone2222222222,
  'Date of birth'44145
}
{
  Email'ccc@cc.com',
  Phone3333333333,
  'Date of birth'40157
}

Data from address sheet

Country'India'State'Karnataka'City'Bangalore' }
Country'India'State'Delhi'City'New Delhi' }
Country'London'State'AAAA'City'BBBB' }



Sunday, July 18, 2021

How to manage Zoom meeting programmatically - PART 2

Part 1 link:  How to manage Zoom meeting programmatically - PART 1

In part 1 of this blog, we understood how to register our application with Zoom to get API Key and Secret Key to be used in programmatic interaction of your application with Zoom services. Now, we will understand how to create, update and delete a zoom meeting.

package.json

{
  "name""node_zoom_exp",
  "version""1.0.0",
  "description""Manage zoom meeting programmatically",
  "main""index.js",
  "scripts": {
    "devStart""nodemon server.js",
    "test""echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [
    "node",
    "express",
    "jsonwebtoken"
  ],
  "author""Jitendra Kumar Singh",
  "license""ISC",
  "dependencies": {
    "axios""^0.21.1",
    "dotenv""^8.2.0",
    "express""^4.17.1",
    "jsonwebtoken""^8.5.1",
    "request""^2.88.2",
    "request-promise""^4.2.6"
  },
  "devDependencies": {
    "nodemon""^2.0.7"
  }
}

Different routes in our node js service

var express = require('express');
const util = require("../utils/util");
const ctrl = require("../controllers/mycontroller");
const auth = require("../middlewares/auth");

var router = express.Router();

router.get('/'ctrl.healthCheck);
router.post('/zoomuserInfo'auth.addTokenctrl.zoomuserInfo);
router.post('/createzoommeeting'auth.addTokenctrl.createZoomMeeting);
router.post('/zoommeeting'auth.addTokenctrl.getMeeting);
router.patch('/updatezoommeeting'auth.addTokenctrl.updateMeeting);
router.delete('/deletezoommeeting'auth.addTokenctrl.deleteMeeting);

module.exports = router;

JWT Token for authentication and authorization on zoom side

In every interaction of our application with Zoom services, we have to send JWT token for authentication and authorization to Zoom service. This JWT token is created by using API Key and Secret key that we have received during registration of our application with zoom market place. I am creating this JWT token in a middleware function in auth.js file

const config = require("../../config");
const jwt = require("jsonwebtoken");

//Use the ApiKey and APISecret from config.js
const payload = {
    iss: config.APIKey,
    exp: ((new Date()).getTime() + 5000)
};
const token = jwt.sign(payloadconfig.APISecret);

function addToken(reqresnext) {
    req.body["token"] = token;
    next();
}

module.exports = { addToken }

I have kept API Key and Secret Key in a config file named as config.js as follows:

const env = process.env.NODE_ENV || 'production'

/**
 * insert your API Key & Secret for each environment 
 * keep this file local and never push it to a public repo for security purposes.
 */
const config = {
    development :{
        APIKey : 'my_apikey_dev_env',
        APISecret : 'my_apisecretkey_dev_env'
    },
    production:{    
        APIKey : 'my_apikey_prod_env',
        APISecret : 'my_apisecretkey_prod_env'
    }
};

module.exports = config[env]

Code for different operation with Zoom meeting

Here, I am giving the code for below operation like 

  • Get details of the user whose email Id is registered with Zoom market place and going to be used as a host email id for all zoom meetings
  • Creating a zoom meeting
  • Update a meeting by changing the meeting time, meeting agenda etc.
  • Delete a meeting
  • Get the details of a meeting
I have added all these code a file mycontroller.js

const SendResponse = require("../utils/sendresponse");
const axios = require('axios');
const sendResponse = new SendResponse();

//Health check function
function healthCheck(reqresnext) {
  try {
    sendResponse.setSuccess(200'Success'"API is running");
    return sendResponse.send(res);
  } catch (error) {
    console.log(error);
  }
}
//Get user detail of host user
async function zoomuserInfo(reqresnext) {
  try {
    const token = req.body.token;
    const email = 'abc@xyz.com'; //host email id
    const result = await axios.get("https://api.zoom.us/v2/users/" + email, {
      headers: {
        'Authorization': 'Bearer ' + token,
        'User-Agent': 'Zoom-api-Jwt-Request',
        'content-type': 'application/json'
      }
    });
    sendResponse.setSuccess(200'Success'result.data);
    return sendResponse.send(res);
  } catch (error) {
    console.log(error.message);
    next();
  }
}
//Create a zoom meeting
async function createZoomMeeting(reqresnext) {
  try {
    const token = req.body.token;
    const email = 'abc@xyz.com'; //host email id;
    const result = await axios.post("https://api.zoom.us/v2/users/" + email + "/meetings", {
      "topic": "Discussion about today's Demo",
      "type": 2,
      "start_time": "2021-03-18T17:00:00",
      "duration": 20,
      "timezone": "India",
      "password": "1234567",
      "agenda": "We will discuss about Today's Demo process",
      "settings": {
        "host_video": true,
        "participant_video": true,
        "cn_meeting": false,
        "in_meeting": true,
        "join_before_host": false,
        "mute_upon_entry": false,
        "watermark": false,
        "use_pmi": false,
        "approval_type": 2,
        "audio": "both",
        "auto_recording": "local",
        "enforce_login": false,
        "registrants_email_notification": false,
        "waiting_room": true,
        "allow_multiple_devices": true
      }
    }, {
      headers: {
        'Authorization': 'Bearer ' + token,
        'User-Agent': 'Zoom-api-Jwt-Request',
        'content-type': 'application/json'
      }
    });
    sendResponse.setSuccess(200'Success'result.data);
    return sendResponse.send(res);
  } catch (error) {
    console.log(error.message);
    next();
  }
}
//Update a zoom meeting
async function updateMeeting(reqresnext) {
  try {
    const token = req.body.token;
    const meetingId = req.body.meetingId;
    const result = await axios.patch("https://api.zoom.us/v2/meetings/" + meetingId, {
      "topic": "UPDATE: Discussion about today's Demo",
      "type": 2,
      "start_time": "2021-03-18T17:00:00",
      "duration": 20,
      "timezone": "India",
      "password": "1234567",
      "agenda": "Discussion about how to update zoome meeting programatically",
      "settings": {
        "host_video": true,
        "participant_video": true,
        "cn_meeting": false,
        "in_meeting": true,
        "join_before_host": false,
        "mute_upon_entry": false,
        "watermark": false,
        "use_pmi": false,
        "approval_type": 2,
        "audio": "both",
        "auto_recording": "local",
        "enforce_login": false,
        "registrants_email_notification": false,
        "waiting_room": true,
        "allow_multiple_devices": true
      }
    }, {
      headers: {
        'Authorization': 'Bearer ' + token,
        'User-Agent': 'Zoom-api-Jwt-Request',
        'content-type': 'application/json'
      }
    });
    sendResponse.setSuccess(200'Success'result.data);
    return sendResponse.send(res);
  } catch (error) {
    console.log(error.message);
    next();
  }
}
//Delete a zoom meeting
async function deleteMeeting(reqresnext) {
  try {
    const token = req.body.token;
    const meetingId = req.body.meetingId;
    const result = await axios.delete("https://api.zoom.us/v2/meetings/" + meetingId, {
      headers: {
        'Authorization': 'Bearer ' + token,
        'User-Agent': 'Zoom-api-Jwt-Request',
        'content-type': 'application/json'
      }
    });
    sendResponse.setSuccess(200'Success'result.data);
    return sendResponse.send(res);
  } catch (error) {
    console.log(error.message);
    next();
  }
}
//Get details of a zoom meeting
async function getMeeting(reqresnext) {
  try {
    const token = req.body.token;
    const meetingId = req.body.meetingId;
    const result = await axios.get("https://api.zoom.us/v2/meetings/" + meetingId, {
      headers: {
        'Authorization': 'Bearer ' + token,
        'User-Agent': 'Zoom-api-Jwt-Request',
        'content-type': 'application/json'
      }
    });
    sendResponse.setSuccess(200'Success'result.data);
    return sendResponse.send(res);
  } catch (error) {
    console.log(error.message);
    next();
  }
}

module.exports = { 
  healthCheckzoomuserInfocreateZoomMeetinggetMeeting
  updateMeetingdeleteMeeting 
}

Thus, we can trigger different services of zoom to perform several operation. We can create recurance meeting too programmatically. We can follow below link to get more information about several zoom services available for programmatica use.

https://marketplace.zoom.us/docs/api-reference/zoom-api/meetings/

Friday, July 16, 2021

How to manage Zoom meeting programmatically - PART 1

 As an user, its very simple to manage the Zoom meeting by signing up in Zoom and managing the meeting through Zoom console. But, as a programmer, we need to integrate Zoom video meeting with our application and we need to manage the Zoom meetings programmatically through our application so that our application's user does not required to be registered with Zoom to use Zoom meeting through our application.

If we want to integrate zoom meeting functionality with our application, we need to follow the below steps:
  1. Either purchase zoom license or we can use free version also but with some limitation like each meeting with only 40 minute max duration, only one meeting at a time for a host, meeting auto recording in Cloud etc. I am explaining here for free version.
  2. Register on Zoom(https://zoom.us/signup) with one of your email id. This email id will be used as host email id while creating the zoom meeting programmatically from your application.
  3. Login to Zoom market place(https://marketplace.zoom.us/) with the credentials generated after signing up in step 2.
  4. Click on the dropdown "Develop" and select "Build App" as shown in below screenshot       

5. In next page, we should select the required App type. There are several options like JWT, OAuth, Chatbot, SDK, Webhook etc. I am going with JWT. So, click on "Create" button in JWT box. I have already created app with JWT, so I am getting "View here" button instead of "Create". 

 6. Fill the form completely and activate your app

7. Finally, we will reach on our app credential page where we will get API Key and API Secret for our app that we registered with Zoom market place. These two keys will be used to create Authentication token programmatically in our application when our application wants to interact with the exposed services of Zoom to manage video meetings


 8. Save these API key and secret key in your environment configuration variables or in DB in your application from where you can get it whenever required to interact with Zoom services.

Important Note: This api key and secret key should be kept very secure in your application. These should not be visible to end user.

Once above all steps are done, we are ready to write code in our application to integrate Zoom meeting feature in our application.

I will add required code(in NodeJs language) to create meeting, update meeting, delete meeting in my next blog.