Fast and Accurate Data Access with Automated QE Solution

The Client

A US-based manufacturer of skincare products with $1.5B annual sales. Millions of customers across the United States, Canada, and Australia exclusively depend on this brand for their skincare and makeup products.

The Challenge

The company’s business is based upon a multi-level marketing model. They needed accurate and timely information on all new sales partner enrollments, processed online via web or mobile apps handled between four geographically dispersed regions.

Upstream systems, implemented in all four regions, use SAP Commerce Cloud (formerly Hybris), an omnichannel e-commerce solution that facilitates customer engagement, product content, and order management, among other features.

Downstream systems use an Apache Solr search implementation and Google Cloud Platform (GCP), including a cloud data warehouse with Big Query tables, a serverless document database that leverages Google Firestore, and cloud storage Buckets. Also, the downstream side uses Salesforce Marketing Cloud for customer support and engagement along with a MuleSoft Secure API Gateway.

Prior to our involvement, the company’s management and administrative staff had to wait for over 45 minutes before getting information on new enrollments. This delay was mainly due to a maze of cron jobs and schedulers that processed and sent data. A related challenge was that the existing infrastructure was simply not up to the task of handling 300K to 400K of streaming data every minute in an accurate manner. The massive volume was produced by the company’s enormous global roster of sales partners that includes updates, orders, and subscriptions.

Before introducing Kafka, all API calls sending data were routed through the MuleSoft API gateway, often causing massive data gridlock. To further compound the problem, the existing architecture did not lend itself to the automated testing and validation of data. The company needed a solution to test the streaming data in all possible combinations promptly and accurately.

The company required:

Fast and accurate access to enrollment data: New partner enrollments were the company’s lifeblood. The company desperately needed quick access to accurate new enrollment data.

The ability to validate and verify a massive amount of data: There was no infrastructure in place to confirm data integrity. Additionally, there was no ability to test data loss, missing mandatory fields from contracts, environment health, flow validation, schema, and formats. The company needed the ability to handle a massive amount of data accurately.

A way to test for possible data flow gridlock: In its original state, the company’s infrastructure had no way to test the streaming data infrastructure for obstructions.

Share this Document

Automation

B2C

Google BigQuery

Google Cloud Platform

Healthcare

MuleSoft

Quality Engineering

Retail

SAP

Related Resources

Empowering Learners with Enhanced Course Discovery and Engagement through ContentHubGPT

Reflecting Excellence: How an Auto Mirror Leader Revved Up Product Descriptions with ContentHubGPT

Reduce Product Time to Market to Just Two Hours with Salsify PXM and ContentHubGPT

The Solution

The company implemented Confluent Cloud, which provided a managed Kafka cloud service. Kafka is a high-speed messaging system that facilitates collecting and routing messages of any type. Confluent provides Kafka management, security, high availability, and elastic scalability.

This approach relieved pressure on the MuleSoft API gateway, resulting in greatly improved performance. Our QE engineers facilitated the company’s seamless migration from an API request-only approach to a streaming-based architecture by incorporating comprehensive validation and verification of real-time data. The overall result was a tremendous improvement in accuracy, allowing the company's managers and administrative staff to process and approve new enrollments almost immediately

For more information on GSPANN's QA automation have a look at Automating Quality of Data Transformations in the Big Data World

The following image summarizes key parts of the new system. In the middle you can see Kafka. To the left are upstream systems, and to the right are downstream systems.

All data flowing from the Hybris / SAP Commerce Cloud and GCE to the respective endpoints, including GCP, the commissions database (GCE), Solr, Salesforce Marketing Cloud, and the MuleSoft API gateway, are now verified automatically under the new system.

Here are a few key points of interest in our solution:

Developed a test automation framework to support Kafka implementation: Augmented Kafka middleware messaging implementation and automated testing to validate and verify data flowing through the system.

Validated contracts in real-time: The new system can validate contracts for new enrollees in real-time, ensuring that the schema is valid and that all mandatory fields contain needed information.

API access to the automation framework: Facilitated access by making several GET request API endpoints available to the company. This solution provides tremendous potential for future growth without developing additional custom software.

Business Impact

Nine times faster response improves productivity: Previously, the company’s management and administrative staff had to wait up to 45 minutes to get the new enrollment information to appear on their systems. The delay period is now reduced to approximately 5 minutes, enhancing productivity and efficiency.

Accurate data leads to a smoother new enrollment process: Before our involvement, the company had little confidence in the produced data. The new testing automation framework validates schema and formats, and the system confirms that all contracts have valid data in mandatory fields. This solution has ensured a smoother flow for the new enrollment process, leading to tremendously improved customer satisfaction.

Consistent data flow and integrity allow IT staff to focus on more important priorities: The new QE automation framework immediately detects data loss, continuously monitors environment health, and ensures data flow and integrity. The company’s IT engineers can now be less concerned over data loss and focus more on other priorities.

A highly scalable platform lays the groundwork for future growth: The new automated testing solution is easily scalable and allows for customer growth without worrying about loss of data or compromising data integrity.

Technologies Used

Java: Popular web-oriented programming languageCucumber: Provides an automated behavior-driven testing infrastructure that facilitates collaboration between business and developmentAllure Framework: Multi-language test reporting toolsApache Solr: Open-source Java-based enterprise search platformSAP Commerce Cloud (formerly Hybris): Supports personalized e-commerce customer engagementGoogle Cloud Platform: Used Firestore, Big Query, and Cloud Storage to provide a serverless document storage data warehouseSalesforce Marketing Cloud: Provides personalized marketing management with deep customer insightsMuleSoft Secure API Gateway: End-to-end API security gateway designed for enterprise-level performanceConfluent Cloud: Provides full featured real-time management for Kafka

Related Capabilities

Utilize Actionable Insights from Multiple Data Hubs to Gain More Customers and Boost Sales

Unlock the power of the data insights buried deep within your diverse systems across the organization. We empower businesses to effectively collect, beautifully visualize, critically analyze, and intelligently interpret data to support organizational goals. Our team ensures good returns on the big data technology investments with the effective use of the latest data and analytics tools.