Usecase Suppose we have a lot of data in our Cloud Storage bucket and somehow by mistake someone runs gsutil rm gs://my_bucket/*, we will lose all our data and won’t be able to recover it easily or may never be able to recover it. How does Object Versioning Help? By Design, every storage object (file) in…

As we all know, By default Docker creates 3 networks automatically Bridge, Host, and None network. Bridge Network The private internal network created by default. Every container is attached to this by default and gets an IP or range 172.17.*.* Containers can also access each other using this IP if required. For…

If you want to work with Airflow and just starting up with your installation then Google Cloud Composer is the best solution, As it creates all the required services and manages Kubernetes Cluster via GKE and everything connects like magic. But if you already have an On-prem Airflow or Airflow…

Few days back I was trying to work with Multiline JSONs (aka. JSON ) on Spark 2.1 and I faced a very peculiar issue while working on Single Line JSON(aka. JSONL or JSON Lines ) vs Multiline JSON files. JSON Lines vs JSON Consider an example, our JSON looks like below here we can see…

Problem — https://leetcode.com/explore/challenge/card/30-day-leetcoding-challenge/530/week-3/3303/ Intuition at every index(i,j) we have to choices either go right or down. Clearly a optimal subproblem solution, first try to implement using recursion, then easily convert to Top-down Dynamic programming. findSum(i,j) = grid[i][j] + min( findSum(i,j+1),findSum(i+1,j)); Solution

In order to understand how your application runs on a cluster, an important thing to know about Dataset/Dataframe transformations is that they fall into two types, narrow and wide, which we will discuss first, before explaining the execution model. Dataframe is nothing but a Dataset[Row], so going forward we will…

Vim is a hell of an editor, which has a very steep learning curve, but very efficient when you are done with it. Starting Editing in Vim editing from command-line $ vi filename open a file inside vim, first entering vim using vi command then use :e command $ vi :e filename Saving file run following…

If you have been using git-bash for command line operations and couldn’t able to find some class paths this blog might help you in adding all class paths permanently to be used from git-bash. Problem Statement You use git-bash a lot and want to access very application from there.(because …

npm audit is a new feature, introduced with npm@6. It shows all vulnerabilities your dependencies got (excluding peerDependencies). You can disable the warning for single package installations with the ‘--no-audit’ flag. Why do we need this ??? If you guys have used Github and have a long running project you might see something like this, hoek@2.16.3…