Deciphering Big Data Reflection

Initial Knowledge & Expectations

Coming into this module, I had practical experience with databases from my banking IT work, but my understanding of "Big Data" was largely theoretical. I knew the 4 Vs framework conceptually but hadn't applied them to real-world data challenges.

My expectation was to gain hands-on experience with Python-based data processing and understand how data governance applies in practice—skills directly relevant to my work at BNI London Branch.

Technical Challenges

The Volume challenge was my first major hurdle. Loading a 1.2GB COVID-19 dataset crashed my initial Jupyter notebook. This forced me to learn chunk processing with Pandas—a technique I now use regularly in production environments.

Understanding when to use SQL vs. NoSQL was another growth area. Through the module's comparative exercises, I developed intuition for choosing the right database paradigm based on data structure and query patterns.

Conceptual Shifts

My biggest mindset shift was around data governance. Initially, I viewed GDPR as a compliance checkbox. Through case study analysis, I came to understand privacy-by-design as a foundational principle that should inform data architecture from the start.

This directly influenced how I approach data projects at work—I now consider data retention, anonymisation, and access controls during the design phase rather than as afterthoughts.

Skills Developed

Professional Application

The skills gained have immediate practical value. I've already applied chunk processing techniques to optimize a monthly reporting script at work, reducing execution time from 45 minutes to 8 minutes.

The data governance frameworks inform my approach to the AML system enhancement project, ensuring we build compliance into the architecture rather than retrofitting it.

Areas for Continued Development

While I've developed strong fundamentals, I want to deepen my expertise in distributed computing frameworks like Apache Spark for truly massive datasets. The module gave me conceptual understanding, but hands-on practice with cluster computing remains a goal.

I'm also keen to explore real-time streaming data processing—building on the velocity concepts from this module to work with technologies like Apache Kafka.

Connection to Future Modules

This module provides essential foundations for the Machine Learning module. Clean, well-structured data is prerequisite for effective model training. The ETL pipeline skills and data quality understanding will directly support my ML project work.