Blog

A data science blog for Petroleum Engineering.

Topics covered

artificial-intelligence artificial-lift batch-automation business-case cloud computational-physics computer-science conference courses data-driven-vs-physics data-engineering data-science data-scientists data-structures datasets deep-learning engineering-library fluid-properties gas-lift geology geophysics geoscience latex linux literate-programming logs loose-questions machine-learning modeling open-source-software paper-research petroleum-engineering petrophysics production-technology prosper pvt python r r-package reproducibility reservoir-engineering reservoir-simulation seismic shiny spe statistics text-mining tikz to-do transcript virtualization visualization-of-data vlp volve webapp well well-data well-logs well-modeling

Subject ▸ data engineering

When Data is Too Big to Sail (in my RAM)

6 October 2017

computer science, data engineering

All of us at some point during our careers had problems with big datasets. So big that our computers could not handle it. Let’s approach the problem from a practical side. I have had similar challenges while working on data science projects with Python and R, and with SQL databases, and with plain vanilla CSV huge files. In one of the cases, the dataset was an Oracle database with well test data from producing oil and gas wells, offshore shallow waters.