Our Work – MySQL and Big Data Stories by Industry

Our Work – Big Data Stories

Automotive I Cable & Telecom I CAD I Computer Repair I Contact Information I Customer Service I E-Commerce I Finance I Gaming I Government I Insurance I Live Chat I Management ConsultingMedical HIPPA Compliance  I Messaging I Mining Industrial I Networking I Phone Apps I Real Estate I Retail I Social Media I Streaming Content I Telecommunications I Telephony I Television I Waste Management I  Web Hosting


Automotive

Automotive Telematics

A major automaker desired to add new telematics features to vehicles, without unnecessarily draining the battery or utilizing excessive, costly cellular data

SOLUTION
Architected and designed custom board-level hardware, vehicle bus protocol, wireless protocol, remotes, and massively scalable database, including evaluating electronic component level vendors and products for inclusion.

TECHNOLOGY USED
CAN bus, ISM-band RF, SMS, LTE, Qualcomm, ARM, UDP/IP, MQTT, C, Rust, Scala, Big Data, Hadoop

INDUSTRY: Automotive

^ back to top


Cable & Telecom

Fiber Optic Broadband Network

Major telecommunications carrier desired to maximize benefits from their fibre network.

SOLUTION
Architected and designed FibreChannel-based network; developed custom pay-per-view and call rating software; developed real-time (50 millisecond) predictive cost minimization telecom switch software; normalized four redundant, heavily text-based build-out locations database into a consolidated, better performingdb; developed real-time (50 millisecond) predictive cost minimization telecom switch software.

TECHNOLOGY USED
Fibre-Channel, C, TSQL, NT, SQL Server, Nortel

INDUSTRY: Cable & Telecom

^ back to top


CAD

Browser-Based CAD Solution

A startup required an easy-to-use yet powerful replacement for their existing AutoCAD workflow.

SOLUTION
Created a columnar, binary, transactional & relational database format with b-tree and quad-tree indexing, implemented clients in Apache Flex and JavaScript / WebGL.

TECHNOLOGY USED
MySQL, Apache Flex, JavaScript, SPA, WebGL, REST

INDUSTRY: CAD

^ back to top

 

Web Delivered CAD Solution

A company that manufactured embroidery machines needed a blended desktop / web solution to replace their legacy desktop MFC app.

SOLUTION
Reverse engineered existing software and formats, created a WebGL-based solution with photo-realistic lighting that worked both in the browser and through Chromium Embedded Framework on the desktop.

TECHNOLOGY USED
C#, ASP.NET, JavaScript, WebGL, Jasmine, require.js, T-SQL, REST, WebSockets, CEF, C++

INDUSTRY: CAD

^ back to top


Computer Repair

Computer Repair Customer Database

An existing company needed to computerize their entire input system, rather than having everything be stored in paper.

SOLUTION
Utilized Webplasm server to create a web-based form to serve as a customer database for previous customer visits, as well as historical repair information. This application eventually became the heart of the point-of-sale system for the repair company.

TECHNOLOGY USED
Webplasm, MySQL

INDUSTRY: E-Commerce, Computer Repair, Retail

^ back to top


Contact Information

Multi-Authority Cloud Data Sync

A company required a system to gather contact data from multiple sources, such as CRMs, mobile devices, and end-user address books. The contact data needed to be combined, transformed and saved, and changes need to be pushed back to the system of origin.

SOLUTION
Created a Storm topology that could handle authentication, rate limits and the intricacies within multiple sources of contact data. The data was transformed into a format that the system could understand. Developed Data regression workflow with data being converted back to the source’s format and sent back.

TECHNOLOGY USED
Storm, Java, REST

INDUSTRY: Social Media, Contact Information

^ back to top


Customer Service

Client Portal with BI Data Lake

A customer chat mobile app company upgraded its enterprise customer facing reporting application and data infrastructure to webscale.

SOLUTION
AgilData developed a real time data lake with data motion from Postgresql into Redshift augmented with a real time data pipe using Alooma (After evaluating Textur and Fivetran) to aggregate five disparate data sources including Google Analytics, Jira, SendGrid and AppsFlyer. Developed Chartio POC with custom dashboards reporting on all datasources and developed embedding strategy for Chartio delivery in iframes within the client portal web container.

TECHNOLOGY USED
Postgresql, Redshift, Textur, Alooma, Fivetran, Google Analytics API, Jira API, SendGrid API, AppsFlyer API, Chartio Embedding

INDUSTRY: Phone Apps, Customer Service, Live Chat

^ back to top


E-Commerce

Enterprise Webserver Project with Custom XML-Based Rendering Language

A startup originally needed a content delivery system for a local record store. Pivoted into an RTSP server to stream music video content. Pivoted into a web server platform.

SOLUTION
Created a full Java web server starting from Java 1.2 through to Java 1.4, complete with NIO stack from the ground up. Wrote a technology stack that handled dynamic rendering of web content using an XML language we created, called Webplasm. At the project’s end, the server was capable of vending and consuming content using SOAP, REST, XML, SDL, RTSP, HTTP, HTTPS, SMTP, and NNTP protocols, all inclusive, with a custom caching stack. Written over the course of 7 years. All code was custom. Two commercial websites used this tech before it was retired in 2009.

TECHNOLOGY USED
Java 1.2 – Java 1.4, JDBC, MySQL, Hypersonic SQL, Java Media Framework (JMF), Subversion

INDUSTRY: Phone App, Customer Service, Live Chat

^ back to top

 

Homegrown Email Drip Campaign System

An e-commerce site needed their own custom Drip Campaign software to allow them to automatically maintain contact with customers.

SOLUTION
Using T-SQL and C# we created a fully customizable system with email templates, response time settings and multiple triggers. This system integrated perfectly with the E-Commerces templates, but could be integrated with any system simply by mapping fields from a table.

TECHNOLOGY USED
C#, T-SQL

INDUSTRY: E-Commerce

^ back to top

 

Computer Repair Customer Database

An existing company needed to computerize their entire input system, rather than having everything be stored in paper.

SOLUTION
Utilized Webplasm server to create a web-based form to serve as a customer database for previous customer visits, as well as historical repair information. This appication eventually became the heart of the point-of-sale system for the repair company.

TECHNOLOGY USED
Webplasm, MySQL

INDUSTRY: E-Commerce, Computer Repair, Retail

^ back to top


Finance

Real-time Closed Captioning Solution

A closed-captioning company with remote workers wanted to replace their existing dial-up system with an internet based one.

SOLUTION
Designed and prototyped a distributed system to route captions in real-time from stenographers home computers, though a central set of servers, to TV stations and web-applications. Innovative push routing allowed for arbitrary network topologies and unexpected failures.

TECHNOLOGY USED
C#, ASP.NET, JavaScript, encryption, protocol analysis, XML, serial communication

INDUSTRY: Television, Financial Reporting

^ back to top

 

Real Estate Valuation Modeling

A major financial company needed a way to increase geographical resolution on their real estate valuation prediction software

SOLUTION
Refactored large amounts of Java and Oracle code to allow for valuation models to be run on ZCTA and Street level data, instead of on County or higher. Segments with missing data could then be inferred from the trends of data around and above it.

TECHNOLOGY USED
Java, Spring, Oracle,

INDUSTRY: Finance, Real Estate

^ back to top

 

Massively-Scalable Identity Resolution System

A startup needed a way to identify duplicate or similar records across a constantly-changing set of billions of records.

SOLUTION
Built a high performance graph-based system using Apache HBase. Graph traversal was pre-computed using a MapReduce-based algorithm similar to Google’s distributed PageRank calculation. The system grew to handle billions of queries per day.

TECHNOLOGY USED
Java, Oracle

INDUSTRY: Finance, Real Estate

^ back to top


Gaming

Yo! Competitor

A startup wanted to create an application that allowed friends to send custom emoticons with captions to each other. Users who did not have devices registered with the competitor could be sent an e-mail with the same emoticon and caption, with a link to the app in the app store, motivating them to download the app.

SOLUTION
Developed a simple, easy to use application for iOS that allowed users to send captions back and forth. Further allowed messages to be sent using push notifications, a simple “inbox” that allowed messages to be stored, and a searchable friends list by e-mail or unique alias. App was in the app store for a couple of short months before being removed due to a trademark dispute. This app is currently being re-designed, and will be re-introduced as “WhatUP!” The app was originally called “HeyU!”

TECHNOLOGY USED
PHP, Apache HTTPD, MySQL, iOS, Push Notification Service, SMTP

INDUSTRY: Gaming, Messaging

^ back to top


Government

Hazardous Waste Disposal Management

A waste disposal company required a system to automate the regulatory compliance processes when customers dispose of hazardous waste. Government systems only accepted paper forms, requiring an automated form generation and signing system.

SOLUTION
Using VBscript and server side Javascript we wrote a form entry system that guided users through the process of filling out the required forms. Forms were generated based on local, state and federal requirements, then a stored signature was applied to the final document, which the users could then print out. The forms were generated based on a large of collection of state and federal rules which were stored in a SQL database.

TECHNOLOGY USED
VBScript, Server Side Javascript, SQL

INDUSTRY: Waste Management, Government

^ back to top


Insurance

Medical Insurance Claims Processing

A startup needed a way to process medical claims from several different insurance providers to create user profiles for analytics.

SOLUTION
Built a roll-your-own ETL solution to import customer claim data. Created an entire backend processing database using MariaDB to process millions of claims using a rules-based processing engine that was developed in house. Co-developed and managed development team in India to work on the front end application, written in Flex.

TECHNOLOGY USED
HTML, Apache Tomcat, MariaDB, Firestorm, dbShards, Apache Flex (Spoon), SQL, Custom Distributed ETL, Custom Rules Engine, Bash script, Git, (J)Ruby, PHP

INDUSTRY: Insurance, Medical, HIPAA Compliant Data Processing

^ back to top


Live Chat

Client Portal with BI Data Lake

A customer chat mobile app company upgraded its enterprise customer facing reporting application and data infrastructure to webscale.

SOLUTION
AgilData developed a real time data lake with data motion from Postgresql into Redshift augmented with a real time data pipe using Alooma (After evaluating Textur and Fivetran) to aggregate five disparate data sources including Google Analytics, Jira, SendGrid and AppsFlyer. Developed Chartio POC with custom dashboards reporting on all datasources and developed embedding strategy for Chartio delivery in iframes within the client portal web container.

TECHNOLOGY USED
Postgresql, Redshift, Textur, Alooma, Fivetran, Google Analytics API, Jira API, SendGrid API, AppsFlyer APi, Chartio Embedding

INDUSTRY: Phone Apps, Customer Service, Live Chat

^ back to top


Management Consulting

Time-shifted Collaboration App in Web-Browser

A collaboration company needed a browser based application that would allow for real-time as well as offline and time-shifted communication, similar to Google Docs, but with the ability to function offline.

SOLUTION
Reviewed the existing pub-sub architecture, reversed engineered the proprietary protocol, created a distributed database solution based around a log-shipping paradigm. Worked with development teams in US, India, and Ukraine to provide on-time delivery.

TECHNOLOGY USED
PostgreSQL, SPA, Jetty, Avro, JavaScript, GWT, Jasmine

INDUSTRY: Management Consulting

^ back to top


Medical HIPPA Compliance Data Compliance

Medical Insurance Claims Processing

A startup needed a way to process medical claims from several different insurance providers to create user profiles for analytics.

SOLUTION
Built a roll-your-own ETL solution to import customer claim data. Created an entire backend processing database using MariaDB to process millions of claims using a rules-based processing engine that was developed in house. Co-developed and managed development team in India to work on the front end application, written in Flex.

TECHNOLOGY USED
HTML, Apache Tomcat, MariaDB, Firestorm, dbShards, Apache Flex (Spoon), SQL, Custom Distributed ETL, Custom Rules Engine, Bash script, Git, (J)Ruby, PHP

INDUSTRY: Insurance, Medical, HIPAA Compliant Data Processing

^ back to top


Messaging

Yo! Competitor

A startup wanted to create an application that allowed friends to send custom emoticons with captions to each other. Users who did not have devices registered with the competitor could be sent an e-mail with the same emoticon and caption, with a link to the app in the app store, motivating them to download the app.

SOLUTION
Developed a simple, easy to use application for iOS that allowed users to send captions back and forth. Further allowed messages to be sent using push notifications, a simple “inbox” that allowed messages to be stored, and a searchable friends list by e-mail or unique alias. App was in the app store for a couple of short months before being removed due to a trademark dispute. This app is currently being re-designed, and will be re-introduced as “WhatUP!” The app was originally called “HeyU!”

TECHNOLOGY USED
PHP, Apache HTTPD, MySQL, iOS, Push Notification Service, SMTP

INDUSTRY: Gaming, Messaging

^ back to top


Mining Industrial

Coal Quality Tracking

A large US coal mining company required a system to deliver quality metrics to their customers for each rail car of coal they shipped.

SOLUTION
Built a customized web application based upon Microsoft technologies, including several cutting edge UI approaches that predated the rise of modern AJAX. Significant background ETL processes were needed to keep the UI database in sync with the customer’s legacy ERP system.

TECHNOLOGY USED
MSSQL, .NET, ASPX JavaScript. ODBC

INDUSTRY: Mining Industrial

^ back to top


Networking

Automated Web-Hosting Infrastructure Management

A web hosting company wanted to automate their operations, in a way that was similar but prior to OpenStack.

SOLUTION
Created a object-oriented set of libraries that could analyze and configure routers, switches, hypervisors, VMs, web servers, and databases. Integrated infrastructure management library with existing web application, automated sign up and tear down with no human intervention.

TECHNOLOGY USED
C#, ASP.NET, JavaScript, SPA, TDD, SQL server, WMI

INDUSTRY: Web Hosting / Networking

^ back to top


Phone Apps

Client Portal with BI Data Lake

A customer chat mobile app company upgraded its enterprise customer facing reporting application and data infrastructure to webscale.

SOLUTION
AgilData developed a real time data lake with data motion from Postgresql into Redshift augmented with a real time data pipe using Alooma (After evaluating Textur and Fivetran) to aggregate five disparate data sources including Google Analytics, Jira, SendGrid and AppsFlyer. Developed Chartio POC with custom dashboards reporting on all datasources and developed embedding strategy for Chartio delivery in iframes within the client portal web container.

TECHNOLOGY USED
Postgresql, Redshift, Textur, Alooma, Fivetran, Google Analytics API, Jira API, SendGrid API, AppsFlyer API, Chartio Embedding

INDUSTRY: Phone Apps, Customer Service, Live Chat

^ back to top


Real Estate

Real Estate Marketing Platform

A startup needed to build a highly-customizable platform for hosting real estate brokerage websites, including individual property websites, agent websites and high quality printed property brochures.

SOLUTION
Built a pluggable MVC-based website framework based on PHP and MySQL. The system integrated deeply with a variety of MLS bureaus, and a flexible ETL scheme was used to unify the various data formats for real estate listing data. A also included one of the industry’s first customized web-based mapping tools that allowed users to search for properties within specific geographic areas, including points of interest, schools, and a variety of other attributes that led toward a fantastic home buying experience for the user.

TECHNOLOGY USED
MySQL, .NET, PHP, JavaScript. ODBC

INDUSTRY: Mining Industrial

^ back to top

 

Real Estate Valuation Modeling

A major financial company needed a way to increase geographical resolution on their real estate valuation prediction software

SOLUTIONRefactored large amounts of Java and Oracle code to allow for valuation models to be run on ZCTA and Street level data, instead of on County or higher. Segments with missing data could then be inferred from the trends of data around and above it.

TECHNOLOGY USED
Java, Spring, Oracle,

INDUSTRY: Finance. Real Estate

^ back to top


Retail

Wholesale Inventory Management and Invoicing System

A London-based jewelry company required a customized back-office system to manage sales orders, inventory, and invoicing.

SOLUTION
Deisnged and built a custom .NET client-server application, backed by Microsoft SQL Server. The application integrated with several back-end data feeds to provide real time VAT calculation, order fulfillment, and customer-facing invoice and shipping reports.

TECHNOLOGY USED
MSSQL, .NET, ASPX JavaScript. ODBC

INDUSTRY: Retail

^ back to top

 

Computer Repair Customer Database

An existing company needed to computerize their entire input system, rather than having everything be stored in paper.

SOLUTIONUtilized Webplasm server to create a web-based form to serve as a customer database for previous customer visits, as well as historical repair information. This appication eventually became the heart of the point-of-sale system for the repair company.

TECHNOLOGY USED
Webplasm, MySQL

INDUSTRY: E-Commerce, Computer Repair, Retail

^ back to top

 

Massively Multiuser Craiglist Competitor

A startup needed a way to post advertisements with attached photos from a mobile device to a large searchable directory. These advertisements served as ways for users to inquire about goods and services listed in the directory, as well as a means for anonymized group communication.

SOLUTION
Rewrote entire system from the ground up using MariaDB and Firestorm. Back-end used web services to handle client communications, front-end application was a mobile app written for iOS. Postings contained a header, sub-header, and a short description with up to 5 pictures posted. Users inquired about postings by sending private messages to the content owners through the app. Created several beta apps, the last of which almost made it to the app store for a launch.

TECHNOLOGY USED
iOS, Firestorm, MariaDB, Linux, Apache Tomcat, Dropwizard, Flurry Analytics, Push Notification Service, CoreData, REST, Java 7

INDUSTRY: Telecommunications, Retail

^ back to top

 

Enterprise Webserver Project with Custom XML-Based Rendering Language

A startup originally needed a content delivery system for a local record store. Pivoted into an RTSP server to stream music video content. Pivoted into a web server platform.

SOLUTION
Created a full Java web server starting from Java 1.2 through to Java 1.4, complete with NIO stack from the ground up. Wrote a technology stack that handled dynamic rendering of web content using an XML language we created, called Webplasm. At the project’s end, the server was capable of vending and consuming content using SOAP, REST, XML, SDL, RTSP, HTTP, HTTPS, SMTP, and NNTP protocols, all inclusive, with a custom caching stack. Written over the course of 7 years. All code was custom. Two commercial websites used this tech before it was retired in 2009.

TECHNOLOGY USED
Java 1.2 – Java 1.4, JDBC, MySQL, Hypersonic SQL, Java Media Framework (JMF), Subversion

INDUSTRY: Retail, Streaming Content, E-Commerce

^ back to top


Social Media

Multi-Authority Cloud Data Sync

A company required a system to gather contact data from multiple sources, such as CRMs, mobile devices, and end-user address books. The contact data needed to be combined, transformed and saved, and changes need to be pushed back to the system of origin.

SOLUTION
Created a Storm topology that could handle authentication, rate limits and the intricacies within multiple sources of contact data. The data was transformed into a format that the system could understand. Developed Data regression workflow with data being converted back to the source’s format and sent back.

TECHNOLOGY USED
Storm, Java, REST

INDUSTRY: Social Media, Contact Information

^ back to top


Streaming Content

Enterprise Webserver Project with Custom XML-Based Rendering Language

A startup originally needed a content delivery system for a local record store. Pivoted into an RTSP server to stream music video content. Pivoted into a web server platform.

SOLUTION
Created a full Java web server starting from Java 1.2 through to Java 1.4, complete with NIO stack from the ground up. Wrote a technology stack that handled dynamic rendering of web content using an XML language we created, called Webplasm. At the project’s end, the server was capable of vending and consuming content using SOAP, REST, XML, SDL, RTSP, HTTP, HTTPS, SMTP, and NNTP protocols, all inclusive, with a custom caching stack. Written over the course of 7 years. All code was custom. Two commercial websites used this tech before it was retired in 2009.

TECHNOLOGY USED
Java 1.2 – Java 1.4, JDBC, MySQL, Hypersonic SQL, Java Media Framework (JMF), Subversion

INDUSTRY: Retail, Streaming Content, E-Commerce

^ back to top


Telecommunications

Massively Multiuser Craiglist Competitor

A startup needed a way to post advertisements with attached photos from a mobile device to a large searchable directory. These advertisements served as ways for users to inquire about goods and services listed in the directory, as well as a means for anonymized group communication.

SOLUTION
Rewrote entire system from the ground up using MariaDB and Firestorm. Back-end used web services to handle client communications, front-end application was a mobile app written for iOS. Postings contained a header, sub-header, and a short description with up to 5 pictures posted. Users inquired about postings by sending private messages to the content owners through the app. Created several beta apps, the last of which almost made it to the app store for a launch.

TECHNOLOGY USED
iOS, Firestorm, MariaDB, Linux, Apache Tomcat, Dropwizard, Flurry Analytics, Push Notification Service, CoreData, REST, Java 7

INDUSTRY: Telecommunications, Retail

^ back to top


Telephony

PBX

Create a VOIP public banch extension server in C#

SOLUTION
Wrote an open-source PBX in C#, which was deployed for a phone company in South Africa.

TECHNOLOGY USED
C#, SIP

INDUSTRY: Telephony

^ back to top


Television

Real-time Closed Captioning Solution

A closed-captioning company with remote workers wanted to replace their existing dial-up system with an internet based one.

SOLUTION
Designed and prototyped a distributed system to route captions in real-time from stenographers home computers, though a central set of servers, to TV stations and web-applications. Innovative push routing allowed for arbitrary network topologies and unexpected failures.

TECHNOLOGY USED
C#, ASP.NET, JavaScript, encryption, protocol analysis, XML, serial communication

INDUSTRY: Television / Financial Reporting

^ back to top


Waste Management

Hazardous Waste Disposal Management

A waste disposal company required a system to automate the regulatory compliance processes when customers dispose of hazardous waste. Government systems only accepted paper forms, requiring an automated form generation and signing system.

SOLUTION
Using VBscript and server side Javascript we wrote a form entry system that guided users through the process of filling out the required forms. Forms were generated based on local, state and federal requirements, then a stored signature was applied to the final document, which the users could then print out. The forms were generated based on a large of collection of state and federal rules which were stored in a SQL database.

TECHNOLOGY USED
VBScript, Server Side Javascript, SQL

INDUSTRY: Waste Management, Government

^ back to top


Web Hosting

Automated Web-Hosting Infrastructure Management

A web hosting company wanted to automate their operations, in a way that was similar but prior to OpenStack.

SOLUTION
Created a object-oriented set of libraries that could analyze and configure routers, switches, hypervisors, VMs, web servers, and databases. Integrated infrastructure management library with existing web application, automated sign up and tear down with no human intervention.

TECHNOLOGY USED
C#, ASP.NET, JavaScript, SPA, TDD, SQL server, WMI

INDUSTRY: Web Hosting / Networking

^ back to top

Top