Completed

Deciphering Big Data

MSc Data Science · University of Essex

12 Units April 2025 cohort · Completed Python, SQL, MongoDB
View Reflection Back to Modules
Learning Journey

Module Units

1

Introduction to Big Data

Understanding the 4 Vs and Common Data Formats

Learning Objectives

  • Identify Big Data characteristics using the 4 Vs framework (Volume, Velocity, Variety, Veracity)
  • Distinguish between different data structures and their use cases
  • Analyze real-world Big Data scenarios using standardized criteria

Key Activities

Completed practical exploration of Big Data characteristics using real-world datasets including COVID-19 global statistics and financial transaction records.

# Chunk processing for large datasets
import pandas as pd

for chunk in pd.read_csv('large_dataset.csv', chunksize=10000):
    process_data(chunk)
Python Pandas Jupyter Notebook
2

Data Collection & Storage

Web Scraping, APIs & Compliance Challenges

Learning Objectives

  • Implement data scraping methods for automated data collection
  • Apply GDPR and data privacy considerations in data collection
  • Develop REST API integration for real-time data access

Key Activities

Built web scraping solutions using BeautifulSoup and Selenium, implemented API integrations with rate limiting and error handling.

BeautifulSoup Selenium REST APIs
3

Data Cleaning & Transformation

Handling Missing Values, Outliers & Normalization

Learning Objectives

  • Apply data cleaning techniques for missing and invalid data
  • Implement outlier detection and handling strategies
  • Transform and normalise data for analysis readiness
4

Database Design & SQL

Entity Relationship Diagrams & Query Optimization

Learning Objectives

  • Design normalized database schemas using ERD methodology
  • Write optimized SQL queries for data retrieval
  • Implement indexing strategies for query performance
5

NoSQL Databases

MongoDB, Document Stores & Schema Flexibility

Learning Objectives

  • Compare SQL and NoSQL database paradigms
  • Implement MongoDB collections and aggregation pipelines
  • Design flexible document schemas for varied data
6

Data Governance & GDPR

Privacy-by-Design, Compliance & Ethics

Learning Objectives

  • Apply GDPR principles to data processing activities
  • Implement data anonymization and pseudonymization
  • Develop data retention and deletion policies

Skills Developed in This Module

Python Pandas SQL MongoDB ETL Pipelines Data Cleaning GDPR Compliance ERD Design Git/GitHub