Consider how one learns as they go through life. As a baby, real-world evidence is immediately experienced physically. Sensory data is consumed voraciously through sight, touch, sound, smell, and taste. The little one’s “big data” starts becoming conceptualised and organised in memory, ready for further analysis and quick action later on.
“Ouch – hot”: don’t touch that.
“Yuck – mashed carrots”: don’t eat that.
Routine memorisation during one’s school-age years is accomplished with a burgeoning creativity in the arts, sciences, sports, and other interests.
Finally, as one matures, full potential is realised from all that data and continuous learning through each experience, culminating with the fulfillment of one’s dreams and aspirations. Ultimately, this enables one to contribute value to themselves and society.
There is a similar progression in data science. Good decisions and productive creativity are based on solid data engineering. That is, the organisation of the facts and experiences enabling the ability to learn from the data and draw the right conclusions. Data is collected, extracted, transformed, and loaded into a form where it can be cleaned, aggregated, and analysed quickly. Then it is applied towards some creative decision or optimisation at the top of the data science hierarchy of needs.
Whether discussing data science or human experience, the ability to engineer data effectively is the foundation upon which creativity and achievement is built. Think of the members of Aerosmith. They couldn’t have written and performed Amazing unless their basic physiological and psychological needs were met. In addition, their mastery and engineering of the foundational “big data” in music, such as chord shapes, riffs, rhythms, scales, and timing were blended with life’s learning experiences and musical expression, in the emotional algorithms of passionate lyrics and poignant melody, to produce an amazing song.
The next time you read or hear about something in data science consider the data engineering involved and the implied relationship between the data science, and Maslow’s, hierarchy of needs. Then remember, there is an amazing data engineer in all of us.