With 22 years of experience in wide range of cloud and client technologies along with deep expertise in BigData Analytics. At McAfee over 9 years, I championed Data Driven Culture. Built data intelligence platform, with that we could scale telemetry and analytics with petabytes of data. Built 7 teams from scratch to build mature data ecosystem. I had been key note speaker on various platform like IEEE, WISE, TechTracks, premium Indian institutes like Sikkim National institute of technology and Jain University. Before that worked with PubMatic a start-up where I could scale infrastructure to handle terabytes of daily data and designed real time bidding system with sub millisecond response. This helped company to compete with Google and revenue reached to 100 million. I worked 8 years with Motorola where in 2006, I introduced Hadoop as early adopter of BigData technology. This enabled voluminous call detail log analysis, to stabilize mobile network.
Testing actions to improve your data quality
Biggest challenge with data ecosystem is to have trust on data and generated insights. To improve trust you need to have high data quality. To achieve high data quality you need to have right data testing plan. Data touches many stages so you need to automate your testing at various stages. You need to focus end to end from client-side data capturing to have data observability in pipe, data quality rules for each event and attributes, data discoverability.