Core Devs | Storage Rental Rate Analysis

Problem

Our client, Nache Nielson, approached us with a considerable challenge. His business required daily tracking of storage rental rates from over 10,000 locations across the United States, sourced from five different company websites. The enormity of this task was depleting his resources and manpower. His goal was to create an efficient solution that would automate this process while allowing for the addition of new location links.

Solution

To fulfill Nache's needs, we built the Stack Storage Web Application from scratch. We initiated custom web scrapers for five unique platforms, garnering daily storage rental data. To manage this data influx, we created a robust backend, facilitating various APIs for user actions. The user interface included a streamlined dashboard, a user-editable profile page, and a flexible graph system for data visualization, providing daily, monthly, or yearly view options.

Challenges We Faced

Throughout the project, we faced multiple hurdles. The main challenge lay in the deployment phase, as we had to manage seven different scripts working in tandem. Other obstacles included building five unique web scrapers for different websites and handling the massive influx of data (10,000+ entries daily) from the scrapers.

How We Solved Them

For deployment, we distributed the seven scripts across two servers: Heroku for frontend and backend, and VPS for the scrapers. This setup ensured smooth operations and a manageable workload. For data handling, we designed a system that would process, filter, and sort the data, making it user-friendly and easily accessible. Despite the unique challenges of every website, our team was successful in building effective scrapers tailored to each one.

Why We Choose This Tech Stack?

Our tech stack was strategically chosen to ensure the smooth development and execution of the Stack Storage Web Application:

Python: We chose Python as our base language because of its simple syntax, versatility, and widespread support for various modules and libraries. It's a great language for scripting, especially for building web scrapers.

Django: Django was selected for creating APIs because of its high scalability, simplicity, and its 'batteries-included' philosophy, which includes necessary features like authentication, routing, and database schema migrations. This allowed us to focus on application development rather than system configuration.

Selenium: Selenium was used for bot development due to its ability to interact seamlessly with different web technologies. This allowed us to scrape data efficiently from various websites which were built using different technologies.

React.js, HTML, CSS, Bootstrap: For frontend development, we utilized this combination due to their incompatibility and the robust, responsive, and interactive user interfaces they can create. React.js helped us build reusable UI components, while Bootstrap offered quick styling solutions.

MongoDB: As a NoSQL database, MongoDB offered us flexibility and scalability in handling and storing our data. Its document-oriented model was ideal for storing large amounts of unstructured data coming from the web scrapers.

Heroku and VPS: Heroku's platform-as-a-service (PaaS) model allowed us to easily deploy and scale our backend and frontend. VPS provided us with dedicated resources to handle the workload of the 5 separate scrapers, ensuring that they worked in unison without any bottlenecks.

More About the Design Process

Our approach to developing the Stack Storage Web Application was systematic and well-structured, ensuring all steps were covered effectively:

Understanding the Requirement

We began by having a thorough discussion with the client to fully understand his needs and challenges. This helped us create a comprehensive project plan.

Creating an MVP

We then created a Minimum Viable Product (MVP) to showcase our proposed solution. This involved outlining the basic structure and functions of the web application.

System Design and Development

Following the MVP approval, we embarked on designing and developing the system. We created five separate web scrapers tailored to the technologies of the different websites from which we needed to extract data.

Backend Development

We constructed a robust backend to manage the massive influx of data. This included functions for data filtering, sorting, and manipulating into an accessible structure. h the Ethereum network and OpenSea platform was a task requiring detailed attention.

API Creation

Simultaneously, we developed APIs for login, adding new locations, editing locations, and more. These APIs allowed for efficient interaction between the client-side and the server-side of the application.

Frontend Development

After setting up the backend, we moved on to the frontend. We focused on creating an intuitive, user-friendly interface featuring a clean dashboard, profile page, and a comprehensive graph system for data visualization.

Testing

Once the development was complete, we thoroughly tested the application for potential bugs and issues. This included stress testing the scrapers and the backend to ensure they could handle the expected loads.

Deployment

Upon successful testing, we deployed the scripts to two different servers: the frontend and backend to Heroku and the five scrapers to VPS servers.

Maintenance and Upgrades

Post-deployment, we have been providing continuous maintenance and regular updates to ensure smooth operation of the application and to add new features as requested by the client.

After meeting all the challenges head-on, we delivered the perfect tool to our client. This case serves as a testament to our capability to tackle complex tasks, delivering solutions that meet our client's needs with precision, quality, and professionalism. ifferent servers: the frontend and backend to Heroku and the five scrapers to VPS servers.

Features

Thanks to our dedicated team, we were able to equip stack Storage with the following features:

Comprehensive Data Collection: Stack Storage is built to collect storage rental rates from over 10,000 different locations across the United States, every day. It automatically scrapes data from five different company websites, enabling users to have a broad and varied analysis of storage rental rates.

Customizable API: Stack Storage offers a customizable API that provides users with the flexibility to adapt the application according to their needs. Users can add or edit locations, manage their profiles, and interact with the data more effectively.

User-friendly Interface: The application comes with a sleek and intuitive dashboard that provides users with an overview of storage rental rates. The profile page allows users to manage their account and preferences, making it easy for them to navigate and control.

Dynamic Data Visualization: The application boasts a flexible graph system for data visualization. Users can choose to view daily, monthly, or yearly data, providing an in-depth understanding of the trends in storage rental rates.

Robust Backend: The backend is designed to efficiently manage the large influx of data. It processes, filters, and sorts the data to ensure it's user-friendly and easily accessible.

Secure Deployment: Stack Storage is deployed on secure servers to ensure the smooth running of the application. The seven scripts are distributed across two servers to manage the workload efficiently.

Real-time Updates: Stack Storage provides users with real-time updates on storage rental rates, ensuring they're always informed and can make data-driven decisions.

Responsive Design: The application is designed to be responsive and works flawlessly on multiple devices, including mobiles, tablets, and desktops. This ensures users can access data and manage their accounts anywhere, anytime.

Advanced Data Management: The application uses MongoDB for data management. This offers flexibility and scalability in handling and storing the large amounts of unstructured data coming from the web scrapers.

Maintenance and Upgrades: The team provides continuous maintenance and regular updates to the application to ensure it remains functional and up-to-date. They also add new features as requested by users to enhance the user experience and provide a more comprehensive service.

Stack Storage stands as a testament to our team's ingenuity, dedication, and ability to transform complex business needs into practical solutions. Its success underscores our prowess in handling vast data and crafting impactful digital tools, reaffirming our commitment to driving business efficiency and success.