Summary Bullets:
• Hewlett Packard Enterprise (HPE) has acquired Pachyderm, a software company that uses reproducible AI to help organizations implement machine learning (ML) projects at scale.
• Pachyderm’s software capabilities will add an important differentiator to HPE’s Machine Learning Development Environment through increased reliability and transparency of the data.
Until not so long ago, Pachyderm was just another independent firm in the crowded space of San Francisco, California’s (US) AI startup landscape. However, many industry players had been aware of Pachyderm for a while.
In 2020, M12, the corporate venture capital subsidiary of Microsoft, helped Pachyderm secure $16 million to expand its ML software platform on account of an impressive record of enterprise adoption by Fortune 500 organizations. Decibel Ventures, an independent venture firm where Cisco is the founding investor, was also a pivotal contributor in that round of investment.
It is not the first time that HPE makes targeted acquisitions of AI startups to ramp up its ML capabilities. The company acquired Determined AI, another San Francisco-based open-source AI firm, in 2021. Determined AI and Pachyderm were both founding members of the AI Infrastructure Alliance, a non-profit foundation that promotes interoperability and collaboration in ML. Their capabilities naturally complement each other and will help boost HPE’s offerings in this market segment.
Pachyderm’s open-source platform for data science and ML, which HPE plans to integrate with its existing supercomputing and AI solutions, helps users automate reproducible ML pipelines for large-scale AI applications. Reproducible AI is the capability to replicate the same results should any parameters change and is crucial for the successful, rigorous application of AI to large enterprise projects.
It follows an essential principle of experimental science: in any research project, the researcher should be able to replicate the experiment, under the same conditions, and achieve the same results. Should conditions change, so should the results. When applied to ML, this means that ML models must be equipped with the technology to reproduce the same results incorporating any circumstantial changes.
Pachyderm’s software capabilities will add an important differentiator to HPE’s Machine Learning Development Environment. By addressing fundamental questions about reliability, reproducible AI helps decision-makers feel more confident about scaling ML projects. Users can repeatedly run an algorithm on certain datasets and obtain the same results on a particular project, building systems that are less prone to errors.
Reproducible AI smooths the path of AI to production, and therefore it is instrumental in AI-at-scale initiatives, because moving ML models from the training stage and into production is one of the biggest stumbling blocks keeping enterprises from effectively leveraging AI. It also embraces current trends in the industry toward greater transparency in AI algorithms, increasing trustworthiness and accuracy in predictions, to meet requirements for greater scrutiny framing the AI and ethics debate.
Pachyderm’s software runs across the major cloud providers and on-premises installations and offers versioning features for ML datasets and a structure similar to GitHub, that facilitates collaboration among data scientists. The company is also known for hosting Pachyderm Hub, a fully managed service with an on-demand compute cluster for AI development.
This acquisition is a smart move by HPE, which is using Pachyderm’s offerings to enhance its AI development platform, to enable users to build and train ML models for popular applications such as computer vision, natural language processing, and data analytics. As complexity grows with AI projects involving larger data sets and permeating vast swathes of the organization in large enterprises, technologies such as reproducible AI will become increasingly sought-after.
IT Connection 