PHP / MySql - Generate Big Reports from Big Data

Big data is a term that describes the large volume of data – both structured and unstructured – that inundates a business on a day-to-day basis. But it’s not the amount of data that’s important. It’s what organizations do with the data that matters. Big data as a service market to grow in near future. “Big Data” analysis is a hot and highly valuable skill now a days. For the big data analysis, an organization needs reports generated from it and to generate reports from big data we need some spacial tricks. In this blog I am going to explain some best practices to generate big reports from big data in PHP / MySql.

Challenges we face in managing Big Data with MySql

In MySql when we have big data we face certain performance issues. Most crucial issue is slow performance of MySql. When you have millions of rows in table and you want to search certain raws from it, it will take some time to get data and you need to improve in that. So in MySql there is a way to make faster data retrieval.

INDEXING

Yes, that 's right indexing solved my issue. There was a table in my database which has 100K records and executing query from it was taking lots of time as there was no indexing. So I created an index on columns which are used mostly for querying that table and after creating index query was working blazing fast.

If you are facing the same issue, better check your database queries and optimize it with indexing to make it faster.

Challenges we face in generating Big Reports with PHP

In PHP when we generate big reports it will surely take time and while working with PHP we have certain restrictions. For example, a PHP script can run for only certain time limits on any server and that limits can not be changed on most of server providers. So in case of generating big reports we surely need time as we have lots of data at backend and from it we are creating reports. So how to deal with it. 

Here is the one solution I always use. You can make use of scheduled cron job. Cron jobs run in background and there is no time restrictions with it so it can run for any amount of time and it can resources as much as possible. So get the reports parameters from the user and save it in database and schedule it for certain time.  For example user want to see day wise sales of products for a month. Save parameters like month in database have a scheduled cron job to read this data and start creating reports in background. Once the report is generated you can notify user about it so they can download it. Which this trick you can generated Big Reports from Big Data.