Subject ▸ Data Engineering

When Data is Too Big to Sail (in my RAM)

All of us at some point during our careers had problems with big datasets. So big that our computers could not handle it. Let’s approach the problem from a practical side. I have had similar challenges while working on data science projects with Python and R, and with SQL databases, and with plain vanilla CSV huge files. In one of the cases, the dataset was an Oracle database with well test data from producing oil and gas wells, offshore shallow waters.