What is data integration?
Data integration is the process of bringing data from separate systems into a form where it can be used together. In practice, this means connecting to databases, files, APIs, and other enterprise sources through purpose-built connectors, then preparing the data for analysis, visualization, or use within an application. Depending on the approach, this can involve techniques such as ETL (Extract, Transform, Load), data replication, or virtualization to consolidate data from any source.
Enterprise data lives across many connected systems, and its value often lies in the relationships among records, systems, and entities. By bringing data into a unified graph model, Tom Sawyer Perspectives helps organizations work directly with those relationships rather than treating each source as a separate silo.
Data connectors bring data from enterprise sources into a unified in-memory graph model.
Considerations when choosing a data integration solution
Selecting the right data integration solution requires balancing business requirements, technical capabilities, performance, scalability, security, and long-term cost. The right solution should support your existing data sources and integration needs while remaining flexible as those requirements evolve.
Consider these points as you evaluate data integration solutions.
Business Objectives and Requirements
Identify what you aim to achieve with the integration, such as improved data quality, real-time analytics, or streamlined operations.
Consider the types of data you need to integrate such as, customer data or financial data, and from which sources (e.g., CRM, ERP).
Data Volume and Complexity
Assess the volume of data that will be handled and the system’s capability to scale as data grows.
Consider the complexity of the data structures and the need for data transformation.
Integration Capabilities
Check if the solution supports different integration styles such as ETL (Extract, Transform, Load), real-time streaming, or batch processing.
Evaluate the ability to connect to various data sources and destinations, including cloud services, databases, and third-party APIs.
Performance and Scalability
Look at the performance benchmarks, particularly how the system performs under load.
Ensure the solution can scale horizontally or vertically based on future needs.
Compliance and Security
Determine the security measures provided, including data encryption and secure data transfer protocols.
Assess data governance capabilities, such as data auditing, lineage, and cataloging.
Ease of Use and Maintenance
Consider the user interface and ease of use for technical and non-technical users.
Evaluate the maintenance support offered by the provider, including customer service, updates, and patches.
Cost
Review the pricing structure, including initial setup costs, licensing fees, and ongoing operational costs.
Consider the total cost of ownership over time, including upgrades and additional services.
Vendor Reputation and Support
Research the vendor’s reputation in the market, their stability, and customer reviews.
Look at the level of technical support provided, including the availability of training resources and community support.
Flexibility and Customization
Check if the solution can be customized to fit specific business needs and how easily these customizations can be implemented.
Assess the flexibility of the solution to adapt to new technologies and integration patterns in the future.
Trial and Testing
If possible, conduct a proof of concept or trial to see how the solution fits with your existing systems and meets your integration needs.Data integration with Tom Sawyer Perspectives
Tom Sawyer Software has spent more than three decades working with data and has a proven track record of providing solutions that address the data silo issue.
Tom Sawyer Perspectives supports multiple data sources and formats, providing the ability to integrate data from a wide range of sources, from graph and relational databases to cloud services, APIs, and more.
Tom Sawyer Perspectives can handle large volumes of data efficiently and scale as your business needs grow.
Once data is integrated, users gain real-time access to a unified view of the data and can use powerful visualization and analysis capabilities to reveal valuable insights.
Supported data integrators
The Tom Sawyer Perspectives platform is data-agnostic. It includes a comprehensive set of data integrators that can populate a model from a data source. Some integrators provide a bidirectional bridge between a data source and a model and support writing model data back to the data source.
Commit configuration
With Tom Sawyer Perspectives, commit configuration gives you control over which changes are committed. By default, automatic bindings for Neo4j, Neptune Gremlin, Neptune openCypher, Kuzu, OrientDB, and TinkerPop integrators handle commit operations automatically. You can also exclude attributes from being committed, and the system supports regular expressions.
Learn more about the graph editing capabilities in Tom Sawyer Perspectives.
The benefits of data commit are clear
Tom Sawyer Perspectives provides the advantages of model persistence while reducing the complexity associated with writing changes back to source systems.
Real-Time Updates
By writing changes directly to the original data source, you can provide real-time updates without delay. This is particularly important in applications where up-to-date information is critical, such as financial systems, real-time monitoring, or collaborative editing tools.
Data Consistency
Writing data back helps maintain data consistency. When you update the data source immediately after modifying it, you ensure that all users or components working with that data see the same changes. This prevents data discrepancies or conflicts.
Historical Tracking
Committing changes in the original data source can provide a complete historical record of all modifications made to the data. This audit trail can be invaluable for debugging, compliance, and accountability purposes.
Scalability
In distributed systems, writing changes back to the original source can distribute the data updates across multiple nodes, improving the scalability and load balancing of your application.
Simplified Recovery
If a commit process fails midway, you can use the recorded changes to recover and reapply updates, ensuring data integrity.
Ease of Collaboration
When multiple users or systems interact with the same data source, writing changes back simplifies collaboration. Everyone sees the same data state, reducing confusion and conflicts.
Reduced Latency
In cases where data retrieval is time-consuming (e.g., fetching data from external APIs or databases), persisting changes locally can reduce latency by avoiding repetitive data fetches.
Easier Rollbacks
If a change leads to unexpected results or errors, reverting to the previous data state is more straightforward when changes are already stored in the original source.
Adherence to Data Policies
In scenarios with strict data governance or compliance requirements, writing changes back to the original data source ensures that data policies and access controls are consistently enforced.
Applying visualization and analysis to your data
Once your integrators are configured in Tom Sawyer Perspectives, the data is loaded into an in-memory native graph model, making it effective and efficient to work with.
Tom Sawyer Perspectives makes it easy to design and configure an end-user web or desktop application that uses the graph model, so users can access data in real time. You can configure easy-to-understand views of the data, including graph drawings, tables, charts, timelines, maps, and more. You can also incorporate powerful analytics into the application so users gain deeper insight into the data.
Watch this video to see how easy it is to configure a dashboard-style layout of views for your Tom Sawyer Perspectives application.
An example application created with Tom Sawyer Perspectives showing a dashboard layout with drawing, map, tree, and chart views.
Query triple stores and labeled property graphs alike without the need to know SPARQL, Gremlin, or Cypher.
Eliminate the need to know complex query languages
You may not know every query that users will want to perform. That is why we created the Pattern Matching Query Builder, which simplifies advanced graph pattern searches without requiring users to know SPARQL, Gremlin, or Cypher. The Pattern Matching Query Builder allows users to search for matching patterns in triple stores and labeled property graphs alike.
However, users who know Cypher or Gremlin can enter their own queries directly. Adding this capability to your Tom Sawyer Perspectives application is seamless for developers, making data exploration more flexible and developer integration easier.
Improve data navigation and analysis with load neighbors
Load neighbors is a feature in Tom Sawyer Perspectives that improves the data navigation and analysis experience for end users. It enables users to explore their data more effectively, saving time and allowing them to focus on their most important tasks.
With load neighbors, users can load data incrementally based on their use case by searching for graph patterns through an intuitive graph visualization. As a result, they can gain insights and make faster decisions.
Load data incrementally and gain insights faster with the Tom Sawyer Perspectives load neighbors feature.