Our Work – Big Data Stories
Automotive I Cable & Telecom I CAD I Computer Repair I Contact Information I Customer Service I E-Commerce I Finance I Gaming I Government I Insurance I Live Chat I Management Consulting I Medical HIPPA Compliance I Messaging I Mining Industrial I Networking I Phone Apps I Real Estate I Retail I Social Media I Streaming Content I Telecommunications I Telephony I Television I Waste Management I Web Hosting
Automotive
Automotive Telematics
A major automaker desired to add new telematics features to vehicles, without unnecessarily draining the battery or utilizing excessive, costly cellular data
SOLUTION
Architected and designed custom board-level hardware, vehicle bus protocol, wireless protocol, remotes, and massively scalable database, including evaluating electronic component level vendors and products for inclusion.
TECHNOLOGY USED
CAN bus, ISM-band RF, SMS, LTE, Qualcomm, ARM, UDP/IP, MQTT, C, Rust, Scala, Big Data, Hadoop
INDUSTRY: Automotive
Cable & Telecom
Fiber Optic Broadband Network
Major telecommunications carrier desired to maximize benefits from their fibre network.
SOLUTION
Architected and designed FibreChannel-based network; developed custom pay-per-view and call rating software; developed real-time (50 millisecond) predictive cost minimization telecom switch software; normalized four redundant, heavily text-based build-out locations database into a consolidated, better performingdb; developed real-time (50 millisecond) predictive cost minimization telecom switch software.
TECHNOLOGY USED
Fibre-Channel, C, TSQL, NT, SQL Server, Nortel
INDUSTRY: Cable & Telecom
CAD
Browser-Based CAD Solution
A startup required an easy-to-use yet powerful replacement for their existing AutoCAD workflow.
SOLUTION
Created a columnar, binary, transactional & relational database format with b-tree and quad-tree indexing, implemented clients in Apache Flex and JavaScript / WebGL.
TECHNOLOGY USED
MySQL, Apache Flex, JavaScript, SPA, WebGL, REST
INDUSTRY: CAD
Web Delivered CAD Solution
A company that manufactured embroidery machines needed a blended desktop / web solution to replace their legacy desktop MFC app.
SOLUTION
Reverse engineered existing software and formats, created a WebGL-based solution with photo-realistic lighting that worked both in the browser and through Chromium Embedded Framework on the desktop.
TECHNOLOGY USED
C#, ASP.NET, JavaScript, WebGL, Jasmine, require.js, T-SQL, REST, WebSockets, CEF, C++
INDUSTRY: CAD
Computer Repair
Computer Repair Customer Database
An existing company needed to computerize their entire input system, rather than having everything be stored in paper.
SOLUTION
Utilized Webplasm server to create a web-based form to serve as a customer database for previous customer visits, as well as historical repair information. This application eventually became the heart of the point-of-sale system for the repair company.
TECHNOLOGY USED
Webplasm, MySQL
INDUSTRY: E-Commerce, Computer Repair, Retail
Contact Information
Multi-Authority Cloud Data Sync
A company required a system to gather contact data from multiple sources, such as CRMs, mobile devices, and end-user address books. The contact data needed to be combined, transformed and saved, and changes need to be pushed back to the system of origin.
SOLUTION
Created a Storm topology that could handle authentication, rate limits and the intricacies within multiple sources of contact data. The data was transformed into a format that the system could understand. Developed Data regression workflow with data being converted back to the source’s format and sent back.
TECHNOLOGY USED
Storm, Java, REST
INDUSTRY: Social Media, Contact Information
Customer Service
Client Portal with BI Data Lake
A customer chat mobile app company upgraded its enterprise customer facing reporting application and data infrastructure to webscale.
SOLUTION
AgilData developed a real time data lake with data motion from Postgresql into Redshift augmented with a real time data pipe using Alooma (After evaluating Textur and Fivetran) to aggregate five disparate data sources including Google Analytics, Jira, SendGrid and AppsFlyer. Developed Chartio POC with custom dashboards reporting on all datasources and developed embedding strategy for Chartio delivery in iframes within the client portal web container.
TECHNOLOGY USED
Postgresql, Redshift, Textur, Alooma, Fivetran, Google Analytics API, Jira API, SendGrid API, AppsFlyer API, Chartio Embedding
INDUSTRY: Phone Apps, Customer Service, Live Chat
E-Commerce
Enterprise Webserver Project with Custom XML-Based Rendering Language
A startup originally needed a content delivery system for a local record store. Pivoted into an RTSP server to stream music video content. Pivoted into a web server platform.
SOLUTION
Created a full Java web server starting from Java 1.2 through to Java 1.4, complete with NIO stack from the ground up. Wrote a technology stack that handled dynamic rendering of web content using an XML language we created, called Webplasm. At the project’s end, the server was capable of vending and consuming content using SOAP, REST, XML, SDL, RTSP, HTTP, HTTPS, SMTP, and NNTP protocols, all inclusive, with a custom caching stack. Written over the course of 7 years. All code was custom. Two commercial websites used this tech before it was retired in 2009.
TECHNOLOGY USED
Java 1.2 – Java 1.4, JDBC, MySQL, Hypersonic SQL, Java Media Framework (JMF), Subversion
INDUSTRY: Phone App, Customer Service, Live Chat
Homegrown Email Drip Campaign System
An e-commerce site needed their own custom Drip Campaign software to allow them to automatically maintain contact with customers.
SOLUTION
Using T-SQL and C# we created a fully customizable system with email templates, response time settings and multiple triggers. This system integrated perfectly with the E-Commerces templates, but could be integrated with any system simply by mapping fields from a table.
TECHNOLOGY USED
C#, T-SQL
INDUSTRY: E-Commerce
Computer Repair Customer Database
An existing company needed to computerize their entire input system, rather than having everything be stored in paper.
SOLUTION
Utilized Webplasm server to create a web-based form to serve as a customer database for previous customer visits, as well as historical repair information. This appication eventually became the heart of the point-of-sale system for the repair company.
TECHNOLOGY USED
Webplasm, MySQL
INDUSTRY: E-Commerce, Computer Repair, Retail
Finance
Real-time Closed Captioning Solution
A closed-captioning company with remote workers wanted to replace their existing dial-up system with an internet based one.
SOLUTION
Designed and prototyped a distributed system to route captions in real-time from stenographers home computers, though a central set of servers, to TV stations and web-applications. Innovative push routing allowed for arbitrary network topologies and unexpected failures.
TECHNOLOGY USED
C#, ASP.NET, JavaScript, encryption, protocol analysis, XML, serial communication
INDUSTRY: Television, Financial Reporting
Real Estate Valuation Modeling
A major financial company needed a way to increase geographical resolution on their real estate valuation prediction software
SOLUTION
Refactored large amounts of Java and Oracle code to allow for valuation models to be run on ZCTA and Street level data, instead of on County or higher. Segments with missing data could then be inferred from the trends of data around and above it.
TECHNOLOGY USED
Java, Spring, Oracle,
INDUSTRY: Finance, Real Estate
Massively-Scalable Identity Resolution System
A startup needed a way to identify duplicate or similar records across a constantly-changing set of billions of records.
SOLUTION
Built a high performance graph-based system using Apache HBase. Graph traversal was pre-computed using a MapReduce-based algorithm similar to Google’s distributed PageRank calculation. The system grew to handle billions of queries per day.
TECHNOLOGY USED
Java, Oracle
INDUSTRY: Finance, Real Estate
Gaming
Yo! Competitor
A startup wanted to create an application that allowed friends to send custom emoticons with captions to each other. Users who did not have devices registered with the competitor could be sent an e-mail with the same emoticon and caption, with a link to the app in the app store, motivating them to download the app.
SOLUTION
Developed a simple, easy to use application for iOS that allowed users to send captions back and forth. Further allowed messages to be sent using push notifications, a simple “inbox” that allowed messages to be stored, and a searchable friends list by e-mail or unique alias. App was in the app store for a couple of short months before being removed due to a trademark dispute. This app is currently being re-designed, and will be re-introduced as “WhatUP!” The app was originally called “HeyU!”
TECHNOLOGY USED
PHP, Apache HTTPD, MySQL, iOS, Push Notification Service, SMTP
INDUSTRY: Gaming, Messaging
Government
Hazardous Waste Disposal Management
A waste disposal company required a system to automate the regulatory compliance processes when customers dispose of hazardous waste. Government systems only accepted paper forms, requiring an automated form generation and signing system.
SOLUTION
Using VBscript and server side Javascript we wrote a form entry system that guided users through the process of filling out the required forms. Forms were generated based on local, state and federal requirements, then a stored signature was applied to the final document, which the users could then print out. The forms were generated based on a large of collection of state and federal rules which were stored in a SQL database.
TECHNOLOGY USED
VBScript, Server Side Javascript, SQL
INDUSTRY: Waste Management, Government
Insurance
Medical Insurance Claims Processing
A startup needed a way to process medical claims from several different insurance providers to create user profiles for analytics.
SOLUTION
Built a roll-your-own ETL solution to import customer claim data. Created an entire backend processing database using MariaDB to process millions of claims using a rules-based processing engine that was developed in house. Co-developed and managed development team in India to work on the front end application, written in Flex.
TECHNOLOGY USED
HTML, Apache Tomcat, MariaDB, Firestorm, dbShards, Apache Flex (Spoon), SQL, Custom Distributed ETL, Custom Rules Engine, Bash script, Git, (J)Ruby, PHP
INDUSTRY: Insurance, Medical, HIPAA Compliant Data Processing
Live Chat
Client Portal with BI Data Lake
A customer chat mobile app company upgraded its enterprise customer facing reporting application and data infrastructure to webscale.
SOLUTION
AgilData developed a real time data lake with data motion from Postgresql into Redshift augmented with a real time data pipe using Alooma (After evaluating Textur and Fivetran) to aggregate five disparate data sources including Google Analytics, Jira, SendGrid and AppsFlyer. Developed Chartio POC with custom dashboards reporting on all datasources and developed embedding strategy for Chartio delivery in iframes within the client portal web container.
TECHNOLOGY USED
Postgresql, Redshift, Textur, Alooma, Fivetran, Google Analytics API, Jira API, SendGrid API, AppsFlyer APi, Chartio Embedding
INDUSTRY: Phone Apps, Customer Service, Live Chat
Management Consulting
Time-shifted Collaboration App in Web-Browser
A collaboration company needed a browser based application that would allow for real-time as well as offline and time-shifted communication, similar to Google Docs, but with the ability to function offline.
SOLUTION
Reviewed the existing pub-sub architecture, reversed engineered the proprietary protocol, created a distributed database solution based around a log-shipping paradigm. Worked with development teams in US, India, and Ukraine to provide on-time delivery.
TECHNOLOGY USED
PostgreSQL, SPA, Jetty, Avro, JavaScript, GWT, Jasmine
INDUSTRY: Management Consulting
Medical HIPPA Compliance Data Compliance
Medical Insurance Claims Processing
A startup needed a way to process medical claims from several different insurance providers to create user profiles for analytics.
SOLUTION
Built a roll-your-own ETL solution to import customer claim data. Created an entire backend processing database using MariaDB to process millions of claims using a rules-based processing engine that was developed in house. Co-developed and managed development team in India to work on the front end application, written in Flex.
TECHNOLOGY USED
HTML, Apache Tomcat, MariaDB, Firestorm, dbShards, Apache Flex (Spoon), SQL, Custom Distributed ETL, Custom Rules Engine, Bash script, Git, (J)Ruby, PHP
INDUSTRY: Insurance, Medical, HIPAA Compliant Data Processing
Messaging
Yo! Competitor
A startup wanted to create an application that allowed friends to send custom emoticons with captions to each other. Users who did not have devices registered with the competitor could be sent an e-mail with the same emoticon and caption, with a link to the app in the app store, motivating them to download the app.
SOLUTION
Developed a simple, easy to use application for iOS that allowed users to send captions back and forth. Further allowed messages to be sent using push notifications, a simple “inbox” that allowed messages to be stored, and a searchable friends list by e-mail or unique alias. App was in the app store for a couple of short months before being removed due to a trademark dispute. This app is currently being re-designed, and will be re-introduced as “WhatUP!” The app was originally called “HeyU!”
TECHNOLOGY USED
PHP, Apache HTTPD, MySQL, iOS, Push Notification Service, SMTP
INDUSTRY: Gaming, Messaging
Mining Industrial
Coal Quality Tracking
A large US coal mining company required a system to deliver quality metrics to their customers for each rail car of coal they shipped.
SOLUTION
Built a customized web application based upon Microsoft technologies, including several cutting edge UI approaches that predated the rise of modern AJAX. Significant background ETL processes were needed to keep the UI database in sync with the customer’s legacy ERP system.
TECHNOLOGY USED
MSSQL, .NET, ASPX JavaScript. ODBC
INDUSTRY: Mining Industrial
Networking
Automated Web-Hosting Infrastructure Management
A web hosting company wanted to automate their operations, in a way that was similar but prior to OpenStack.
SOLUTION
Created a object-oriented set of libraries that could analyze and configure routers, switches, hypervisors, VMs, web servers, and databases. Integrated infrastructure management library with existing web application, automated sign up and tear down with no human intervention.
TECHNOLOGY USED
C#, ASP.NET, JavaScript, SPA, TDD, SQL server, WMI
INDUSTRY: Web Hosting / Networking
Phone Apps
Client Portal with BI Data Lake
A customer chat mobile app company upgraded its enterprise customer facing reporting application and data infrastructure to webscale.
SOLUTION
AgilData developed a real time data lake with data motion from Postgresql into Redshift augmented with a real time data pipe using Alooma (After evaluating Textur and Fivetran) to aggregate five disparate data sources including Google Analytics, Jira, SendGrid and AppsFlyer. Developed Chartio POC with custom dashboards reporting on all datasources and developed embedding strategy for Chartio delivery in iframes within the client portal web container.
TECHNOLOGY USED
Postgresql, Redshift, Textur, Alooma, Fivetran, Google Analytics API, Jira API, SendGrid API, AppsFlyer API, Chartio Embedding
INDUSTRY: Phone Apps, Customer Service, Live Chat
Real Estate
Real Estate Marketing Platform
A startup needed to build a highly-customizable platform for hosting real estate brokerage websites, including individual property websites, agent websites and high quality printed property brochures.
SOLUTION
Built a pluggable MVC-based website framework based on PHP and MySQL. The system integrated deeply with a variety of MLS bureaus, and a flexible ETL scheme was used to unify the various data formats for real estate listing data. A also included one of the industry’s first customized web-based mapping tools that allowed users to search for properties within specific geographic areas, including points of interest, schools, and a variety of other attributes that led toward a fantastic home buying experience for the user.
TECHNOLOGY USED
MySQL, .NET, PHP, JavaScript. ODBC
INDUSTRY: Mining Industrial
Real Estate Valuation Modeling
A major financial company needed a way to increase geographical resolution on their real estate valuation prediction software
SOLUTIONRefactored large amounts of Java and Oracle code to allow for valuation models to be run on ZCTA and Street level data, instead of on County or higher. Segments with missing data could then be inferred from the trends of data around and above it.
TECHNOLOGY USED
Java, Spring, Oracle,
INDUSTRY: Finance. Real Estate
Retail
Wholesale Inventory Management and Invoicing System
A London-based jewelry company required a customized back-office system to manage sales orders, inventory, and invoicing.
SOLUTION
Deisnged and built a custom .NET client-server application, backed by Microsoft SQL Server. The application integrated with several back-end data feeds to provide real time VAT calculation, order fulfillment, and customer-facing invoice and shipping reports.
TECHNOLOGY USED
MSSQL, .NET, ASPX JavaScript. ODBC
INDUSTRY: Retail
Computer Repair Customer Database
An existing company needed to computerize their entire input system, rather than having everything be stored in paper.
SOLUTIONUtilized Webplasm server to create a web-based form to serve as a customer database for previous customer visits, as well as historical repair information. This appication eventually became the heart of the point-of-sale system for the repair company.
TECHNOLOGY USED
Webplasm, MySQL
INDUSTRY: E-Commerce, Computer Repair, Retail
Massively Multiuser Craiglist Competitor
A startup needed a way to post advertisements with attached photos from a mobile device to a large searchable directory. These advertisements served as ways for users to inquire about goods and services listed in the directory, as well as a means for anonymized group communication.
SOLUTION
Rewrote entire system from the ground up using MariaDB and Firestorm. Back-end used web services to handle client communications, front-end application was a mobile app written for iOS. Postings contained a header, sub-header, and a short description with up to 5 pictures posted. Users inquired about postings by sending private messages to the content owners through the app. Created several beta apps, the last of which almost made it to the app store for a launch.
TECHNOLOGY USED
iOS, Firestorm, MariaDB, Linux, Apache Tomcat, Dropwizard, Flurry Analytics, Push Notification Service, CoreData, REST, Java 7
INDUSTRY: Telecommunications, Retail
Enterprise Webserver Project with Custom XML-Based Rendering Language
A startup originally needed a content delivery system for a local record store. Pivoted into an RTSP server to stream music video content. Pivoted into a web server platform.
SOLUTION
Created a full Java web server starting from Java 1.2 through to Java 1.4, complete with NIO stack from the ground up. Wrote a technology stack that handled dynamic rendering of web content using an XML language we created, called Webplasm. At the project’s end, the server was capable of vending and consuming content using SOAP, REST, XML, SDL, RTSP, HTTP, HTTPS, SMTP, and NNTP protocols, all inclusive, with a custom caching stack. Written over the course of 7 years. All code was custom. Two commercial websites used this tech before it was retired in 2009.
TECHNOLOGY USED
Java 1.2 – Java 1.4, JDBC, MySQL, Hypersonic SQL, Java Media Framework (JMF), Subversion
INDUSTRY: Retail, Streaming Content, E-Commerce
Social Media
Multi-Authority Cloud Data Sync
A company required a system to gather contact data from multiple sources, such as CRMs, mobile devices, and end-user address books. The contact data needed to be combined, transformed and saved, and changes need to be pushed back to the system of origin.
SOLUTION
Created a Storm topology that could handle authentication, rate limits and the intricacies within multiple sources of contact data. The data was transformed into a format that the system could understand. Developed Data regression workflow with data being converted back to the source’s format and sent back.
TECHNOLOGY USED
Storm, Java, REST
INDUSTRY: Social Media, Contact Information
Streaming Content
Enterprise Webserver Project with Custom XML-Based Rendering Language
A startup originally needed a content delivery system for a local record store. Pivoted into an RTSP server to stream music video content. Pivoted into a web server platform.
SOLUTION
Created a full Java web server starting from Java 1.2 through to Java 1.4, complete with NIO stack from the ground up. Wrote a technology stack that handled dynamic rendering of web content using an XML language we created, called Webplasm. At the project’s end, the server was capable of vending and consuming content using SOAP, REST, XML, SDL, RTSP, HTTP, HTTPS, SMTP, and NNTP protocols, all inclusive, with a custom caching stack. Written over the course of 7 years. All code was custom. Two commercial websites used this tech before it was retired in 2009.
TECHNOLOGY USED
Java 1.2 – Java 1.4, JDBC, MySQL, Hypersonic SQL, Java Media Framework (JMF), Subversion
INDUSTRY: Retail, Streaming Content, E-Commerce
Telecommunications
Massively Multiuser Craiglist Competitor
A startup needed a way to post advertisements with attached photos from a mobile device to a large searchable directory. These advertisements served as ways for users to inquire about goods and services listed in the directory, as well as a means for anonymized group communication.
SOLUTION
Rewrote entire system from the ground up using MariaDB and Firestorm. Back-end used web services to handle client communications, front-end application was a mobile app written for iOS. Postings contained a header, sub-header, and a short description with up to 5 pictures posted. Users inquired about postings by sending private messages to the content owners through the app. Created several beta apps, the last of which almost made it to the app store for a launch.
TECHNOLOGY USED
iOS, Firestorm, MariaDB, Linux, Apache Tomcat, Dropwizard, Flurry Analytics, Push Notification Service, CoreData, REST, Java 7
INDUSTRY: Telecommunications, Retail
Telephony
PBX
Create a VOIP public banch extension server in C#
SOLUTION
Wrote an open-source PBX in C#, which was deployed for a phone company in South Africa.
TECHNOLOGY USED
C#, SIP
INDUSTRY: Telephony
Television
Real-time Closed Captioning Solution
A closed-captioning company with remote workers wanted to replace their existing dial-up system with an internet based one.
SOLUTION
Designed and prototyped a distributed system to route captions in real-time from stenographers home computers, though a central set of servers, to TV stations and web-applications. Innovative push routing allowed for arbitrary network topologies and unexpected failures.
TECHNOLOGY USED
C#, ASP.NET, JavaScript, encryption, protocol analysis, XML, serial communication
INDUSTRY: Television / Financial Reporting
Waste Management
Hazardous Waste Disposal Management
A waste disposal company required a system to automate the regulatory compliance processes when customers dispose of hazardous waste. Government systems only accepted paper forms, requiring an automated form generation and signing system.
SOLUTION
Using VBscript and server side Javascript we wrote a form entry system that guided users through the process of filling out the required forms. Forms were generated based on local, state and federal requirements, then a stored signature was applied to the final document, which the users could then print out. The forms were generated based on a large of collection of state and federal rules which were stored in a SQL database.
TECHNOLOGY USED
VBScript, Server Side Javascript, SQL
INDUSTRY: Waste Management, Government
Web Hosting
Automated Web-Hosting Infrastructure Management
A web hosting company wanted to automate their operations, in a way that was similar but prior to OpenStack.
SOLUTION
Created a object-oriented set of libraries that could analyze and configure routers, switches, hypervisors, VMs, web servers, and databases. Integrated infrastructure management library with existing web application, automated sign up and tear down with no human intervention.
TECHNOLOGY USED
C#, ASP.NET, JavaScript, SPA, TDD, SQL server, WMI
INDUSTRY: Web Hosting / Networking
