Big data is a term used for data sets that are very large or complex where traditional data processing application software is inadequate to handle processing the scale of information quickly and cost-effectively. Big data challenges include capturing data, data storage, data analysis, search, sharing, transfer, visualization, querying, updating and information privacy.

Here are some examples:

Facebook – Uses big data to process all the posts, likes, pictures and more for many years. How do they know who to connect you to? Targeted Ads? Who’s most likely to be single soon? These are all answered with big data.

World Health Organization – Uses big data to pull information from all over the world to help prevent diseases and understand where to allocate resources and drive impact.

Capital One – Leverages big data to understand their customer’s spending habits so they can better tailor their products to real needs.

City of Long Beach – Uses big data to find people illegally watering their lawns during non-sanctioned times which provides compliance and saves water.

The most common technology solution to solve big data processing is Hadoop; which takes the classic data processing paradigm but looks at it from a more standard software development mindset.  The classic way people manage data is utilizing large database systems that centralize the processing of this data into large monolithic systems like Oracle or SQL Server or Teradata.  The issue is running a large monolithic database application to manage and process data requires meticulous and time-consuming structuring can be difficult and expensive to scale at large sizes.  Hadoop moved away from structured data and created a distributed model to process data more horizontally, distributed to any number of computers or nodes.  The raw processing power is across a large number of nodes and the processing is broken up so the jobs can be run in smaller chunks across a truly distributed system.

Organizations large and small are recognizing that they can make smarter decisions by processing and analyzing all the data they have. In today’s information economy data has evolved into a minable asset. These assets need to be analyzed in real time because small changes in the data have to be understood instantly. By leveraging the power of big data, companies can now look at much larger datasets with greater speed, providing insights that deliver value and competitive advantage.

Some examples would be:

– Determining operational efficiencies across large technology installations using log data.  

– Determining all customer interactions across many large websites.  

– Gathering disease data from all over the world.  

– Understanding who is invested in blockchain technology.  Analyzing worldwide patent data to determine innovation trends.  

– Understanding why we have a worldwide obesity epidemic.

– Sharing and understanding scientific research data.

– Understanding customer buying patterns.

Blockchain and big data share many similarities.  Blockchain at its core is simply a distributed ledger.  Any transaction needs to be validated and verified.  When you make a credit card purchase the card issuer validates that you have enough available credit and eventually transfer these funds to the seller.  This is an example of a centralized ledger.  The bank controls this information and controls the decision-making process.  This is fine if the market always wants to rely on large expensive organizations to do this.  What if I am just a small website selling goods and I want to transact in a very cheap way that cuts out these people in the middle all taking a fee.  I could earn more for myself and get paid instantly.  This is what something like bitcoin can provide.  The bitcoin network consists of individual nodes or miners processing these transactions for a very small fee paid to them in bitcoin.  The distributed nature also makes this very secure as no one node has all the transaction data.  They each process a block of the info and then when all blocks are processed the transaction is approved and money is transferred.  There is no settlement process either so people get their money immediately.  This allows for global scale transaction processing at low cost that’s also highly secure.

Ethereum takes the distributed ledger concept a step further and allows true business logic to sit on top of the distributed ledger allowing for intelligence to be added into the process.  Using the website example above let’s pretend the website sells software development work and the payout structure depends on meeting certain milestones.  If you were using a standard transaction processing distributed ledger like bitcoin, you would have to take on the task of building all this logic into your website to manage and track all the work and the milestones and potentially manually manage the payments.  Using Ethereum you could build all of this logic into what’s called a smart contract.  This smart contract has the logic needed to manage all of this so you don’t have to.  You have to do the development to manage the smart contract but once that’s done the whole process is now automated.  This ability has taken the whole blockchain concept to a new level.

While big data is very popular now it still requires a fair amount of technical expertise and expense to implement.  Big data is one of the hottest skills right now and hiring or contracting is expensive.  Running Big data at scale also requires significant technical capabilities.  Large companies with deep pockets can do this, but not everyone can.  That’s the problem we are solving. We are democratizing big data.

It’s  possible to create a very small managed big data instance that can run on nodes on computers globally.  The ability to send a big data job into a decentralized big data environment creates infinite scale and lowers the cost to run these jobs dramatically.  The benefit for those running these nodes is payment in BDB tokens.  Our intuitive interface allows jobs to be submitted with just a few clicks.  Very little technical knowledge is needed to run these previously complex and expensive jobs and you only pay when you need a job run.  All that is needed is the data, the metadata about this data, and the rules for analyzing the data.  The job is run on as many nodes as are available and returned to the requestor.

Another significant advantage BDB is overall data security as data is spread across many nodes and all the data is destroyed after each job is run leaving no trail and no data to be hacked.

Sara runs a company that owns and operates a small solar farm to provide energy to her local area.  This farm collects a lot of log data from all the solar panels.  She is a small energy producer so every dollar counts and she wants to be sure her solar panels are as optimized as possible.  She is collecting all this log data but doesn’t have a large tech staff nor does she have the experience to parse it all and gain any real insight from it. Here’s where BDB comes in.   With BDB Sara can now take all the data and create performance ranges from highest to lowest of the energy output of each of her solar panels.  This will immediately tell her which panels are the best and worst performing.  From this data she can then look into the individual logs for the worst performing and see if there are obvious indications of any issue with those panels.  

Sara simply logs into the BDB portal and defines her BDB job.   Once the job is set up the system will determine the size and complexity and provide a cost in BDB’s and Sara can decide if she would like to execute the job.  She executes this job at a cost of $38 USD.  The data is then uploaded to the BDB system which processes and sends the output file back to our secure landing zone for Sara to let her know her job is done!  Sara decides to run this job weekly and pays less than $155 per month!