What Size Data Can Auto Insights Handle?
The recommended file size for Auto Insights is up to 10GB. For our typical use cases, like procurement, finance, sales, complaints and workforce analytics, we find that the data can vary in size from 10MB to 5GB.
For example, in a 10GB dataset, a procurement function spending between $20 to $50 billion a year would be able to capture 3 to 4 years worth of transactions easily (assuming the data had approximately 20 columns and 80 to 100 million rows).
Where your .csv file is greater than 2GB in size, we recommend zipping the file before uploading to Auto Insights using 7zip, which is free for Windows. It’s absolutely not necessary to zip your file, but it does make the upload process faster!
What if my data is more than 10GB?
If your data is more than 10GB, we suggest you speak to our Business Solutions team for a consultative session where we can explore techniques such as aggregation or other techniques to cater to your needs.
What if I have terabytes (TB) of data?
Common examples we see of terabytes of data are:
The lowest level, most detailed granularity of transactions for all customers in a loyalty program of large supermarket chain over 3 years (for example, Customer 1 buys 1 apple in CBD store at $2 on 18 March 2019).
The lowest level, most detailed granularity of transactions of all credit card holders for the Big 4 Australian banks (or similar size) over 2 to 3 years (for example, Customer 1 spends $50 on Uber Eats on 18th March 2019 at 8 pm).
In this case, our business solution team can offer a consultative session where we can explore techniques such as aggregation or other techniques to cater to your needs.
What if I have petabytes (PB) of data?
YouTube 4K videos (all videos globally)
Facebook content (all content globally)
We don’t analyze unstructured data such as Facebook content or videos in their raw format. However, logs or transactions like user behavior can be uploaded to Auto Insights if the data is in a structured data format, which will significantly reduce the data size. Again, our Business Solutions team can offer a consultative session where we can explore techniques such as aggregation or other techniques to cater to your needs.