Automated market price analysis
Summary
Meesenburg Großhandel KG was founded already 250 years ago. In recent years, the company has already lived through three industrial revolutions and has always adapted to market conditions. In this way, Meesenburg is also actively facing up to the current changes in digitalization and sustainability. The automated market price analysis enables the company to offer market-oriented prices to its partners and customers.
The challenge
Online trading offers customers a way to quickly get an overview of the current market prices of a product and compare offers from different retailers. In order to still be able to convince its customers to buy from Meesenburg, the company strives for market-oriented pricing through an internal comparison platform, in addition to many services and customer benefits.
The steadily growing number of online retailers and the increasing affinity of customers to use digital and automated procurement options present the company with new challenges in pricing.
In order to adequately assess the market, a digital platform is needed to quickly and efficiently compare market prices. For a realistic assessment of Meesenburg’s specific assortment, an internal comparison platform is to be created based on Meesenburg’s assortment data.
The solution
In forming market prices, the first step was to evaluate the sources from which the information could be obtained. This was limited to publicly available data sources. One challenge here was that the information from these data sources was unstructured and played out in the form of different UIs. In order to display the desired information in a normalized way in an internal platform, source-specific web crawlers were implemented, running serverless in an AWS Fargate cluster. The crawlers search the sources at regular intervals and extract the desired information via the HTML DOM tree.
One challenge here was the highly individual structures of each public source. In addition to the direct search of an EAN, complete category trees were also read in to index all products of a source. Some sources are their own marketplaces, which also aggregate the prices of different merchants as well as play out graduated prices. The collected data was processed, normalized, and stored in an AWS DynamoDB.
To serve the requirement that the data should be provided via an API (API First Approach), but in addition should be made available to the employees in a visual format, a One Page Application was implemented with the Angular Framework and provided via a REST API through an API Gateway. The business logic is completely mapped in AWS Lambdas (Function as a Service).
The result
In the course of this project, a web platform was created for internal use, which makes it possible to compare the prices of different sources in a time-serialized manner. For this purpose, the various scale prices as well as the individual offers from the sources were recorded and structured. ChartJS was used to visually prepare this information and provide it as interactive charts.
In addition, all information from this comparison platform can be accessed for programmatic use via a REST API. The watchlist of products can be administered via the API.
An intuitive comparison platform was created that allows employees to compare current prices and obtain reference values for the formation of prices. In addition, serialized data collection makes it possible to identify trends and make forecasts.
The comparison platform is run as a serverless application on AWS.
Contact
